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Why GAO Did This Study 

The No Child Left Behind Act of 
2001 (NCLBA) requires districts 
with schools that have not met 
state performance goals for 
3 consecutive years to offer their 
low-income students supplemental 
educational services (SES), such as 
tutoring, if these schools receive 
Title I funds. SES are provided 
outside of the regular school day by 
a state-approved provider, with 
responsibility for implementation 
shared by states and districts. GAO 
examined (1) how SES 
participation changed between 
school years 2003-2004 and 2004- 
2005; (2) how SES providers are 
working with districts to deliver 
SES; (3) how states are monitoring 
and evaluating SES; and (4) how 
the Department of Education 
(Education) monitors and supports 
state implementation of SES. To 
collect data on SES, GAO surveyed 
all states and a nationally 
representative sample of districts 
with schools required to offer SES. 
We also visited 4 school districts, 
interviewed 22 SES providers, 
reviewed SES-related research, and 
interviewed Education staff. 



What GAO Recommends 



GAO recommends that Education 
disseminate information on 
promising practices used to 
improve SES implementation, 
provide states with technical 
assistance to improve evaluation of 
SES’s effect on student 
achievement, and expand program 
flexibility where appropriate. 
Education generally supported 
GAO’s recommendations. 

www.gao.gov/cgi-bin/getrpt7GAO-06-758. 

To view the full product, including the scope 
and methodology, click on the link above. 

For more information, contact Mamie Shaul at 
(202) 512-7215 orshaulm@gao.gov. 



What GAO Found 

SES participation among eligible students increased from 12 to 19 percent 
between school years 2003-2004 and 2004-2005, and the number of recipients 
also increased, due in part to a rise in the number of schools required to offer 
services. Districts have used some promising practices to inform parents and 
encourage participation, such as offering services on school campuses and at 
various times. However, challenges remain, including timely and effective 
notification of parents and attracting providers to serve certain areas and 
students, such as rural districts or students with disabilities. 

To promote improved student academic achievement, SES providers took steps 
to align their curriculum with district instruction and communicate with 
teachers and parents, though the extent of their efforts varied. A majority of the 
22 providers we interviewed worked to align SES and district curriculum by 
hiring teachers familiar with the district curriculum as tutors. However, at least 
some providers did not have any contact with teachers in about 40 percent of 
districts. Both providers and district officials experienced challenges related to 
contracting and coordination of service delivery. Providers, districts, and 
schools reported that greater involvement of schools would improve SES 
delivery and coordination, as it has in some places where this is occurring. 

While state monitoring of district and provider efforts to implement SES has 
been limited in past years, more states reported conducting on-site reviews and 
other monitoring activities during 2005-2006. In addition, districts have taken a 
direct role in monitoring providers, and their monitoring efforts have similarly 
increased. Although states are required to withdraw approval from providers 
that fail to increase student academic achievement for 2 years, many states 
struggle to develop meaningful SES evaluations. While a few states have 
completed evaluations, none provides a conclusive assessment of SES providers’ 
effect on student academic achievement. 

Several Education offices monitor SES activity across the country and provide 
SES support to states and districts through written guidance, grants, and 
technical assistance. However, states and districts reported needing additional 
SES evaluation support and technical assistance. For example, 85 percent of 
states reported needing assistance with methods for evaluating SES. Many also 
voiced interest in Education’s pilot programs that increase SES flexibility, 
including the one that allowed certain districts identified as in need of 
improvement to act as providers. 

SES Is Often Delivered after School and on School Campuses 
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August 4, 2006 
Congressional Requesters 

Title I of the No Child Left Behind Act (NCLBA) supports the acadenuc 
achievement of 16.6 million low-income students nationwide by providing 
them with additional opportimities, such as supplemental educational 
services (SES), if their schools are low performing. SES includes tutoring 
and remediation that are provided outside of the regular school day by a 
state-approved provider, such as a for-profit company or a conummity- 
based organization. When a Title I school does not meet state performance 
goals for 2 years, including goals set under the Elementary and Secondary 
Education Act (ESEA) before the enactment of NCLBA, the district must 
offer students the choice of transferring to another school in the district 
that is not in improvement status. When a Title I school does not meet 
state performance goals for 3 or more years, the district must offer SES to 
all of the low-income students enrolled in the school. Districts with 
schools required to offer school choice and SES must set aside an amount 
equal to 20 percent of their Title I funds to provide choice-related 
transportation and SES for eligible students in these schools. In school 
year 2006-2006, $12.7 billion in federal Title I funding was distributed to 
nearly all school districts and approximately half of the public schools 
nationwide in order to improve the education of low-income students. 

While states set NCLBA performance goals and schools are judged on the 
performance of their students, responsibihty for SES implementation is 
primarily shared by states and districts under the law. States are 
responsible for approving SES providers to serve students in their states 
and, therefore, review provider applications to assess each provider’s 
record of effectiveness and program design, including its instructional 
strategies and service costs. States are also responsible for monitoring 
SES providers and evaluating their services. Districts are responsible for 
notifying parents of their child’s eligibility for SES and contracting with 
the state-approved providers parents select for services. At the federal 
level, the U.S. Department of Education (Education) oversees SES 
implementation nationwide and provides guidance and technical 
assistance. 

Although some districts were first required to offer SES in 2002-2003, 
others did not have to offer SES until 2003-2004 or the year after, and 
therefore, states and districts are at different stages of implementing the 
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SES provisions. Further, research suggests that many face challenges to 
increasing student participation, working with providers to ensure 
students receive quality services, and monitoring and evaluating these 
services. In response to congressional interest in these and other issues 
related to early implementation of SES, and in anticipation of 
reauthorization of the ESEA, we agreed to answer the following questions: 
(1) How has SES participation changed between school years 2003-2004 
and 2004-2005, and what actions have been taken to increase 
participation? (2) How are providers working with districts and schools to 
provide services that increase student achievement? (3) To what extent 
are states monitoring and evaluating SES implementation? (4) How does 
Education monitor state SES implementation and assist state and district 
efforts? 

To answer our four research questions, we collected data through several 
methods. First, to gather state-level information on SES, we collected 
information through a Web-based survey of state SES coordinators in all 
50 states, the District of Columbia, and Puerto Rico. Further, to gather 
district-level information on SES, we conducted a mail survey of district 
SES coordinators from a nationally representative sample of districts with 
schools required to offer SES. To ensure that we obtained information 
from the largest districts with schools required to offer SES, we included 
all 21 districts with 100,000 or more enrolled students in our sample. In 
designing our surveys, we reviewed survey questions used by Education. 
The district and state surveys were both administered between January 
and March 2006. Seventy-seven percent of district SES coordinators, 
including all coordinators from districts with 100,000 or more enrolled 
students, and all state SES coordinators responded to the surveys. All 
percentage estimates presented from the district survey have margins of 
error of plus or minus 8 percentage points, unless otherwise noted. While 
we did not validate specific information that states and districts reported 
through our surveys, we reviewed the information to determine that their 
responses were complete and reasonable and found the information to be 
sufficiently reliable for the purposes of this report. To gather federal-level 
information on SES implementation, we interviewed Education officials 
and analyzed Education’s data on SES available from state reports and 
ongoing studies. 

To collect additional information from states and districts, as well as 
information from schools and providers, we conducted site visits to four 
school districts that had experience with SES implementation (Woodbum, 
Ore.; Newark, N.J.; Chicago, 111.; and Hamilton County, Tenn.). During the 
visits, we spoke with state and district officials, as well as officials from a 
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total of 12 schools and 15 providers. When viewed as a group, the site visit 
districts provided variation across characteristics such as geographic 
location, district size, student ethnicity, and the percentage of students 
with limited English proficiency or disabilities. In addition to the 
15 providers we spoke with during the site visits, we also interviewed 
7 for-profit SES providers operating in multiple states, for a total of 
22 providers. In our surveys and other data collection efforts, we asked 
questions about SES implementation during specific school years, and, 
therefore, all years cited refer to school years. 

See appendix 1 for detailed information on both of our surveys as well as 
our other data collection methods. We conducted our work from August 
2005 through July 2006 in accordance with generally accepted government 
auditing standards. 



Results in Brief 



Nationally, SES participation increased from 12 percent of eligible 
students receiving services in 2003-2004 to 19 percent in 2004-2005, but 
challenges to increasing participation remain. The total number of 
students receiving services rose from approximately 117,000 in 2002-2003 
to 430,000 in 2004-2005, due in part to a rise in the number of schools 
required to offer services. An estimated 20 percent of districts required to 
offer SES had no students receiving services, and the majority of those 
districts were rural or had less than 2,500 enrolled students. While these 
districts, therefore, spent no funds for SES, some other districts that were 
providing services reported needing more than the 20 percent Title 1 set- 
aside to serve all eligible students that had requested services. While a 
student’s eligibility for SES is based on income and not academic 
achievement, most students receiving services were among the lower 
achieving students in school. Districts have taken multiple actions to 
encourage participation, such as using several methods to contact parents 
and offering services on school campuses and at various times. For 
example, 90 percent or more of districts provided parents with written 
information, held individual meetings with parents, encouraged school 
staff to talk with parents about SES, and offered services in locations 
easily accessible to students, such as on or near the school campus. 
Although some districts have used promising practices to increase SES 
participation, notifying parents in a timely and effective way continues to 
be a challenge for districts. About half of the districts did not notify 
parents before the begiiming of the current school year, due in part to 
delays in receipt of school improvement results from their state. Further, 
several providers indicated that confusing parental notification letters and 
enrollment forms may discourage participation. States and districts have 
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also been challenged to ensure there are providers to serve students from 
rural areas or students with limited English proficiency or disabilities. 
Encouraging student attendance has also been a challenge, in part because 
students may participate in other after-school activities, such as sports or 
work. 

Providers reported that they are working with districts and schools to 
deliver quality services, but contracting with districts and coordinating 
with schools remain challenges, and all parties reported that 
implementation would be improved with greater school involvement. To 
help improve student academic achievement, providers align their 
curriculum with district instruction primarily by hiring teachers familiar 
with the district curriculum and communicating with the teachers of 
participating students. Providers reported communicating with teachers in 
person as well as mailing information and progress reports to them, which 
are similar to the methods reported for communicating with parents. 
However, we estimate that some, most, or all providers did not contact 
teachers in about 40 percent of districts and did not contact parents in 
about 30 percent of districts during the 2004-2005 school year. In addition, 
contracting for services was sometimes a challenge for providers and 
districts. Some of the providers we interviewed said certain districts 
impose burdensome contract requirements, such as limiting provider 
marketing and use of school facilities. About 40 percent of districts also 
considered contracting to be a challenge, and officials in three of the four 
districts we visited expressed concerns about their lack of authority under 
the law to set parameters around costs and program design, such as the 
number of service hours per student and the tutor-to-student ratio. While 
officials from one state we visited expressed interest in additional federal 
guidance on setting such parameters, another state helped to address 
some of its district’s concerns and improve transparency and 
accountability by requiring approved providers to submit cost information 
and sharing this information with districts. About 70 percent of the states 
also reported that coordination among providers, districts, and schools is 
a moderate to very great challenge. For example, services were delayed or 
withdrawn in three of the four districts we visited because not enough 
students signed up to meet the providers’ enrollment targets and districts 
were not aware of these targets. Because SES may be delivered in school 
facilities, providers and officials in the districts and schools we visited 
reported that involvement of school administrators and teachers can ease 
SES delivery and coordination with providers. 

Prior to 2005-2006, state SES monitoring of both districts and providers 
was limited, and over the last few years, many states have struggled to 
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develop meaningful SES evaluations. In regard to monitoring districts, 
more states reported conducting on-site reviews in 2005-2006, and many 
states also collected information from other sources, such as school 
principals and parents. Further, nearly all states were collecting 
information on districts’ expenditures for SES and their efforts to notify 
parents of available services. In regard to monitoring providers, though 
states were challenged by their oversight capacity, many states reported 
using a more active monitoring approach in 2005-2006. For example, 
approximately three-fourths indicated they were conducting on-site 
reviews, and many were also collecting information from districts, 
schools, and parents to monitor SES providers in that year. In addition to 
states, districts have also taken an active role in monitoring providers, and 
similarly increased their efforts in 2005-2006. While states are required to 
withdraw approval from providers that fail to increase student academic 
achievement for 2 years, many are struggling to develop meaningful SES 
evaluations that assess this outcome. Approximately three-fourths of 
states reported that they are experiencing challenges evaluating SES, 
including designing methods for determining sufficient academic progress 
of students, having the time and knowledge to analyze SES data, and 
developing data systems to track SES information. For example, during 
our site visits to Illinois and New Jersey, state officials noted that they 
were currently in the process of improving their data collection systems to 
more effectively capture and analyze data for SES evaluations. At the time 
of our survey, only a few states had drafted or completed an evaluation 
report addressing SES providers’ effect on student academic achievement, 
and no state had produced a report that provided a conclusive assessment 
of this effect. In addition, over half of the states reported that they were in 
the process of conducting an SES evaluation in order to meet the federal 
requirement. 

Several Education offices monitor SES and support implementation 
through written guidance and technical assistance grants, but states and 
districts cite the need for additional assistance and flexibility. Two 
Education offices are primarily responsible for SES monitoring. The Office 
of Innovation and Improvement (Oil) monitors SES implementation by 
reviewing SES-related reports and responding to state, district, and 
provider concerns brought to its attention, while the Office of Elementary 
and Secondary Education (OESE) visits states and districts as part of its 
overall Title I monitoring. The Office of the Inspector General (OIG) also 
conducts monitoring through on-site SES audits. Education shares 
monitoring results internally and has used them to help states and districts 
address implementation challenges. For example. Oil has updated the SES 
non-regulatory guidance several times since 2002, and OESE has 
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recommended actions needed to improve state compliance with federal 
regulations. Education also issues grants that provide SES implementation 
support. For example, OESE recently funded the Center on Innovation and 
Improvement, which provides technical assistance to regional centers to 
help them work with states to improve Education’s programs, including 
SES. Despite Education’s efforts, many states and districts reported that 
they need additional information or assistance with certain aspects of SES 
implementation to better comply with SES requirements. For example, 

85 percent of states and an estimated 70 percent of districts needed 
assistance with methods for evaluating SES. Further, states and districts 
both indicated needing additional capacity to implement SES. Officials 
also reported needing additional or clearer guidance from Education on 
certain SES provisions, such as crafting a notification letter that is both 
complete and easy for parents to understand. Although Education has 
cited several states for districts’ incomplete notification letters, the 
department’s current SES guidance provides a sample parental 
notification letter that does not clearly specify all of the required elements. 
Many states also reported needing a forum to share promising practices 
with peers, such as meetings and conferences. In addition, many states 
and districts expressed interest in expansion of Education’s pilot 
programs. For example, some officials suggested expanding Education’s 
pilot that enables low-achieving districts typically restricted from being 
SES providers to provide these services on the condition that they 
evaluate the effect of their services on student achievement. Officials 
noted this may ease access to SES and potentially reduce costs. 

To help states and districts implement SES more effectively, we are 
recommending that Education collect and disseminate information on 
promising practices used by districts, including examples of how districts 
have successfully involved school officials in SES implementation, and 
sample parental notification letters that meet federal requirements and are 
easy for parents to understand. Further, to improve states’ and districts’ 
ability to provide services to the maximum number of students, we are 
recommending that Education consider expanding its current SES pilot 
program allowing selected districts in need of improvement to serve as 
providers and clarify state authority to set parameters around service 
design and costs. To improve federal and state monitoring of SES, we are 
recommending that Education require states to collect and submit 
information on the amount and percent of Title I funds spent on SES by 
districts and provide states with technical assistance and additional 
guidance on how to evaluate the effect of SES on student academic 
achievement. 
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In written comments on a draft of this report, Education supported our 
recommendations and cited actions the department has already initiated 
or plans to take in addressing them. Education also provided technical 
comments, which were incorporated into the report as appropriate. 


Background 


Under NCLBA, SES primarily includes tutoring provided outside of the 
regular school day that is designed to increase the academic achievement 
of economically disadvantaged students in low-performing Title I schools. 
These services must consist of high-quality, research-based instruction 
that aligns with state educational standards and district curriculum. 


Title I 


Title T of ESEA, as amended and reauthorized by NCLBA, authorizes 
federal funds to help elementary and secondary schools establish and 
maintain programs that will improve the educational opportunities of 
economically disadvantaged children. Title I is the largest federal program 
supporting education in kindergarten through 12th grade, supplying 
$12.7 billion in federal funds in fiscal year 2006. According to Education, 
during the 2005-06 school year, nearly all U.S. school districts and 
approximately half of public schools received some Title I funding. In 
addition, the latest national data available from Education counted 
16.5 million students as Title I participants in the 2002-2003 school year. 




Title I funds are distributed by formula to state education agencies, which 
retain a share for administration and school improvement activities before 
passing most of the funds on to school districts. Districts are required to 
distribute Title I funds first to schools with poverty rates over 75 percent, 
with any remaining funds distributed at their discretion to schools in rank 
order of poverty either districtwide or within grade spans. A school’s Title 
I status can change from year to year because school enrollment numbers 
and demographics may vary over time. 


NCLBA 


Enactment of NCLBA strengthened accountability by requiring states and 
schools to improve the academic performance of their students so that all 
students are proficient in reading and math by 2014. Under NCLBA, each 
state creates its own content standards, academic achievement tests, and 



^ In this report, we refer to Title I, Part A of ESEA as “Title I.” Other Parts of Title I (Parts B, 
C, and D) are targeted at specific populations or purposes and are commonly referred to by 
their program names, such as Even Start. 
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proficiency levels. In 2005-2006, states were required to test all children for 
reading and mathematics achievement annually in grades 3-8 and once in 
high school to determine whether schools are making adequate yearly 
progress (AYP).^ 

In addition to meeting the state’s performance goals by grade, subject, and 
overall student population, schools are responsible for meeting those 
goals for designated groups. These groups are students who (1) are 
economically disadvantaged, (2) are part of a racial or ethnic group that 
represents a significant proportion of a school’s student population, 

(3) have disabilities, or (4) have limited English proficiency. To make AYP, 
each school must also show that each of these groups met the state 
proficiency goals for both reading and math. In addition, schools must 
show that at least 95 percent of students in grades required to take the test 
have done so. Schools must also demonstrate that they have met state 
targets for at least one other academic indicator, including graduation rate 
in high schools and a state-selected measure in elementary or middle 
schools.^ 

For Title I schools that do not meet state AYP goals, NCLBA requires the 
implementation of specific interventions, and these interventions must 
continue until the school has met AYP for 2 consecutive years. Table 1 
outlines the interventions applied after each year a Title I school misses 
state performance goals. At their discretion, states may also implement 
interventions for public schools that do not receive Title I funds and do 
not make AYP. Although districts are not required to offer SES until a Title 
I school has missed performance goals for 3 years, because some schools 
had not met state goals set under ESEA before the enactment of NCLBA, 
some Title I schools were first required to offer SES in 2002-2003, the first 
year of NCLBA implementation. 



^ State testing has been phased in, whereby from 2002-2004 states were required to 
administer tests at least once in grades 3-6, 6-9, and 10-12. Beginning in 2005-2006, states 
must administer annual tests in grades 3-8, and at least once in grades 10-12. Testing for 
science proficiency will begin in 2007-2008. 

^ Measures for elementary and middle schools may include, but are not limited to, 
attendance, grade-to-grade retention rates, or changes in the percentage of students 
completing gifted and talented programs. 
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Table 1: NCLBA Interventions for Schools Not Meeting Yearly Performance Goals 
over Time 


Number of years 
school misses 
performance goals 


School status in the next 
year 


NCLBA interventions for Title I 
schools 


First year missed 


N/A 


None 


Second year missed 


Needs Improvement - 
First Year 


Required to offer school choice 


Third year missed 


Needs Improvement - 
Second Year 


Required to offer school choice 
and SES 


Fourth year missed 


Corrective Action® 


Required to offer school choice 
and SES 


Fifth year missed 


Planning for Restructuring" 


Required to offer school choice 
and SES 


Sixth year missed 


Implementation of 
Restructuring 


Required to offer school choice 
and SES 



Source: GAO analysis of NCLBA. 



Note: N/A = not applicable. 

“Corrective action is a significant intervention in a school that is designed to remedy the school's 
persistent inability to make adequate progress toward all students becoming proficient in reading and 
mathematics. 

“Restructuring is a major reorganization of a school, involving fundamental reforms, such as 
significant changes in the school’s staffing and governance. For example, some schools may be 
converted to charter schools during restructuring. 



States are also required to establish and implement AYP standards for 
school districts based on the performance of ah of the schools in the 
district. If districts fail to meet these standards for 2 consecutive years, 
states may classify districts as needing improvement. A district identified 
for improvement must develop and implement an improvement plan and 
remain in this status until it meets AYP standards for 2 consecutive years. 
If a district remains in improvement status for 2 or more years, it may be 
identified for corrective action as deemed necessary by the state. 



SES under NCLBA Students are eligible for SES if they attend Title 1 schools that have missed 

AYP for 3 consecutive years and are from low-income families. School 
districts must determine family income on the same basis they use to 
make allocations to schools under Title 1, for which many have historically 
used National School Lunch Program (NSLP) data. The NSLP is a federally 
funded program that annually collects family income data from students’ 
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parents to determine student eligibility for free and reduced-priced lunch/ 
A student’s state assessment scores, grades, and other academic 
achievement information are generally not considered when determining 
SES eligibility. However, if sufficient funds are not available to provide 
SES to all eligible children, school districts must give priority to the 
lowest-achieving eligible students. 

SES providers may include nonprofit entities, for-profit entities, school 
districts, public schools, public charter schools, private schools, public or 
private institutions of higher education, educational service agencies, and 
faith-based organizations. Under the Title 1 regulations that govern SES, a 
district identified as in need of improvement or corrective action may not 
be an SES provider, though its schools that are not identified as needing 
improvement may. In addition, individual teachers who work in a school 
or district identified as in need of improvement may be hired by any state- 
approved provider to serve as a tutor in its program. 

A district must set aside an amount equal to 20 percent of its Title 1 
allocation to fund both SES and transportation for students who elect to 
attend other schools under school choice. This set-aside cannot be spent 
on administrative costs for these activities, and the district may reallocate 
any unused set-aside funds to other Title 1 activities after ensuring all 
eligible students have had adequate time to opt to transfer to another 
school or apply for SES. Funding available for SES is, therefore, somewhat 
dependent on costs for choice-related transportation, though as we found 
in our 2004 report on NCLBA’s school choice provisions, few students are 
participating in the school choice option.^ If a district does not incur any 
choice-related transportation costs, it must use the full 20 percent set- 



* In addition, NSLP allows certain schools and districts with high percentages of low- 
income families to certify students as eligible for free and reduced price lunches about 
once every 4 years. These alternatives to the traditional requirements for annual 
certification, known as “Provision 2” and “Provision 3,” reduce local paperwork and 
administrative burden. Education has determined that, for purposes of identifying students 
as “economically disadvantaged” for SES, school officials may deem all students in 
Provision 2 and Provision 3 schools as “economically disadvemtaged.” See policy memo: 
U.S. Department of Agriculture, Food and Nutrition Service, “No Child Left Behind Act 
(NCLB),” School Meals (Washington, D.C.: Feb. 2003), 
http://www.fns.usda.gOv/end/govemance/policy-memos/2002-02-20.htm 

^ Only 1 percent of students ehgible for school choice elected to transfer to another school 
during the 2003-2004 school year. See GAO, No Child Left Behind Act: Education Needs to 
Provide Additional Technical Assistance and Conduct Implementation Studies for 
School Choice Provision, GAO-05-07 (Washington, D.C.: Dec. 10, 2004). 
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aside amount to pay for SES if sufficient demand for services exists. In 
addition, if the Title I set-aside is not sufficient to fund SES for interested 
students, both states and districts may direct other funds for these 
services at their discretion. For each student receiving SES, a district must 
spend an amount equal to its Title I per-pupil allocation or the actual cost 
of provider services, whichever is less.'’ 

Education oversees SES implementation by monitoring states and 
providing technical assistance and support. Oil leads SES policy 
development and coordinates the publication of SES guidance, and OESE 
oversees and monitors Title I, including SES. NCLBA and the Title I 
regulations and SES guidance outline the roles and responsibilities states, 
school districts, parents, and service providers have in ensuring that 
eligible students receive additional academic assistance through SES 
(see table 2). 



® A district calculates the Title I per pupil allocation by dividing its total Title I, Part A 
allocation by the number of children residing within the district aged 5-17 who are from 
families below the poverty level, as determined by the most recent Census Bureau 
estimates from the Department of Commerce. 
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Table 2: SES Stakeholder Roles and Responsibilities 


Stakehoider 


Roles and responsibilities 


State 


Set criteria and standards for approving providers 
Identify, approve, and maintain public list of providers 

Ensure that the list of approved providers includes organizations that are able to serve students with disabilities and 
limited English proficiency 

Monitor and evaluate the effectiveness of provider services 
Monitor district SES implementation 

Develop and use policy criteria for withdrawing providers from state-approved list, including if 

• provider fails for 2 consecutive years to increase student proficiency relative to state academic content and 
achievement standards 

• provider fails to adhere to applicable health, safety, and civil rights requirements 


School district 


Provide an annual notice to parents, which must identify available providers, describe the enrollment process and 
timeline, describe the services, qualifications, and demonstrated effectiveness of each provider, and be easily 
understandable 

Help parents choose a provider, if requested 

Protect the privacy of students eligible for and receiving services 

Calculate and establish the SES per pupil allocation If not determined by the state 

Determine which students should receive services if more students apply for SES than can be served with available 
funds 

Enter into contracts with providers 

Ensure eligible students with disabilities and eligible students with limited English proficiency may participate in SES 

At the discretion of the state, may be involved in collecting data from providers to assist state monitoring and 
evaluation activities 


Providers 


Provide services in accordance with district agreements 
Enable students to attain their individual achievement goals 

Measure student progress and inform parents and teachers of progress made by students 
Ensure non-disclosure of student data to the public 

Provide services consistent with applicable health, safety, and civil rights laws 
Provide services that are secular, neutral, and non-ideological 


Parents 


Choose a provider from the state-approved list 

Are encouraged to be actively involved In their child’s SES program 



Source: GAO, per P.L107-110, 34 C.F.R. Part 200, or the U.S. Department of Education, Supplemental Educational Services Non- 
Regulatory Guidance, June 2005. 



During the 2005-2006 school year, Education announced the 
implementation of two pilot programs intended to increase the number of 
eligible students receiving SES and generate additional information about 
the effectiveness of SES on students’ academic achievement. In the first, 
Education permitted four districts in Virginia to offer SES instead of 
school choice in schools that are in their first year of needs improvement. 
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In the second, Education entered into flexibility agreements with the 
Boston and Chicago school districts, enabling them to act as SES 
providers while in improvement status. Oil and OESE coordinated 
implementation of the pilots for the department. Both pilots were subject 
to review at the end of the 2005-2006 school year, at which time Education 
planned to evaluate their effect on student academic achievement. 



SES Participation Has 
Increased as Districts 
Have Taken Steps to 
Ease Access, but 
Challenges Remain 



SES participation increased between 2003-2004 and 2004-2005, and most 
students receiving services were among the lower achieving students in 
school. Districts have taken multiple actions to encourage participation, 
such as offering services on or near the school campus or at various times, 
Despite these efforts, challenges to increasing participation remain, 
including notifying parents in a timely and effective manner, ensuring 
there are providers to serve certain areas and students, and encouraging 
student attendance. 



Participation among Nationally, the participation rate increased substantially from 12 percent 

Eligible Students of eligible students receiving SES in 2003-2004 to 19 percent in 2004-2005.^ 

Increased from 12 to 19 addition, the number of students receiving services almost quadrupled 

Percent between 2003-2004 SOOS/"'' 2 « 04 - 20«5 from approximately 117^)00 to 430,000 

students nationwide, based on the best available national data (see fig. 1). 
This increase may be due in part to the increase in the number of schools 
required to offer SES over that time period. Specifically, between 2004- 
2005 and 2005-2006 the number of schools required to offer SES increased 
from an estimated 4,509 to 6,584.® 



and 2004-2005, and Most 
Participants Were Low 
Achieving 



’’ Certain states did not submit SES recipient information to Education through their 
NCLBA Consolidated State Performance Reports for all years. See notes to figure 1 for 
details. Further, Education did not collect data on the number of students eligible for SES 
in 2002-2003, and therefore an estimate of the SES participation rate is unavailable for that 
year. 

® In addition to our analysis, the Center on Education Policy reported that within districts 
required to offer SES, the percentage of all schools required to offer SES rose from 27 
percent of schools in 2002-2003 to 65 percent of schools in 2005-2006. See the Center on 
Education Policy, From the Capital to the Classroom, Year 4 of the No Child Left Behind 
Act (Washington D.C.: March 2006), for more information. 
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Figure 1 : Number of Students Receiving SES Nationwide (2002-2003 to 2004-2005) 



Number of students 
500,000 



450,000 430,044 




School year 

Source: Education’s NCLBA Consolidated State Performance Reports and GAO state survey. 

Note: Certain states did not submit SES recipient information to Education through their NCLBA 
Consolidated State Performance Reports for all years. Specifically, 2002-2003 data from Kansas and 
North Dakota, 2003-2004 data from Pennsylvania, and 2004-2005 data from New Jersey are not 
included in the figure. In addition, 2002-2003 data from New York only include information from New 
York City. 

Although nationally SES participation is increasing, some districts 
required to offer SES have no students receiving services. Specifically, we 
estimate that no students received services in about 20 percent of the 
approximately 1,000 districts required to offer SES in 2004-2005. A 
majority of these districts were rural or had a total enrollment of fewer 
than 2,500 students. Our survey did not provide sufficient information to 
explain why these districts had no students receiving services in 2004- 
2005; therefore, it is unclear whether their lack of participation was related 
to district SES implementation or other issues. 

Nationwide, we estimate that districts required to offer SES spent 
the equivalent of 5 percent of their total Title 1 funds for SES in 2004-2005 
excluding administrative expenditures. Districts set aside an amount equal 
to 20 percent of their Title 1 funds for SES and choice-related 
transportation at the beginning of the school year, and the proportion of 
the set-aside spent on SES varied by district. Specifically, in 2004-2005, 
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about 40 percent of districts spent 20 percent or less of the set-aside to 
provide SES and almost one-fifth of districts spent over 80 percent.'’ 
Nationwide, of the total amount districts set-aside for SES, we estimate 
they spent 42 percent on SES, excluding administrative expenditures. 

Further, an estimated 16 percent of districts reported that the required 
Title 1 set-aside was not sufficient to fund SES for all eligible students 
whose parents requested services. For example, during our site visit to 
Newark, N.J., district officials reported budgeting the entire 20 percent 
Title 1 set-aside to fund SES in 2004-2005, but with this amount of funding, 
the district was only able to fund SES for 17 percent of the students 
eligible for services. In addition, according to Chicago, 111., district 
officials, the district budgeted the entire 20 percent Title I set-aside to fund 
SES in 2005-2006, and because parents’ demand for services significantly 
exceeded the amount of funding available, the district also allocated 
$5 million in local funds to provide SES. 

While approximately 1,000 of the over 14,000 districts nationwide were 
required to offer SES in 2004-2005, SES recipients are concentrated in a 
small group of large districts, as 56 percent of recipients attended school 
in the 21 districts required to offer SES with more than 100,000 total 
enrolled students (see fig. 2). Further, states ranged from having 0 districts 
to 257 districts required to offer SES in 2004-2005, with most states having 
fewer than 10 districts required to offer SES. State differences in the 
number of districts required to offer SES may have resulted from 
differences in performance or differences in state proficiency standards 
and methods used to measure adequate yearly progress. 



® These district estimates have a margin of error that exceeds plus or minus 8 percentage 
points. See table 9 in appendix 1 for more information. 
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Figure 2: School Districts Required to Offer SES in 2004-2005 




■ other districts required to offer SES 

Districts required to offer SES with over 1 00,000 totai enroiied students 



Source: GAO. 



Students receiving SES in 2004-2005 shared certain characteristics, as 
districts reported that most students receiving services were among the 
lower achieving students in school. Specifically, an estimated 91 percent 
of the districts that reviewed the academic records of students receiving 
SES classified most or all of the students receiving SES as academically 
low achieving.^" For example, Hamilton County, Tenn., school officials said 



^°We did not review the academic achievement records of students receiving SES or 
independently verify this information obteimed through the district survey. 
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that students receiving SES are frequently behind grade level in their skills 
and require special attention to increase their academic achievement. 

Further, we estimate that over half of SES recipients were elementary 
school students in the majority of districts and about 60 percent of schools 
required to offer SES in 2004-2005 were elementary schools.'^ Districts 
varied in the percentage of students with limited English proficiency 
receiving services. In about one-third of districts, less than 5 percent of 
SES recipients were students with limited English proficiency; however, in 
about one-fifth of districts, over half of SES recipients were students with 
limited English proficiency. Students with disabilities made up less than 
20 percent of students receiving services in about two-thirds of districts. 
Finally, in some districts, the majority of SES recipients were African- 
American or Hispanic. In about 40 percent of districts, over half of SES 
recipients were African-American, and in about 30 percent of districts, 
over half of SES recipients were Hispanic. Because we were unable to 
obtain comparable data on the characteristics of Title 1 students enrolled 
in these districts in 2004-2005, we were unable to determine whether 
certain groups of students were underserved. 

We estimate that about 2,800 providers delivered services to students 
nationwide in 2004-2005, and more providers were available to deliver 
services in the districts with the largest student enrollments. Specifically, 
about 80 percent of districts had between 1 and 5 providers delivering 
services in 2004-2005. However, the number of providers delivering 
services in the 21 districts with more than 100,000 total enrolled students 
ranged from 4 to 45, and averaged 15 providers per district in 2004-2005. 



Many of the district estimates included in this paragraph have a margin of error that 
exceeds plus or minus 8 percentage points. See table 9 in appendix I for more information. 

In addition to our analysis, the Center on Education Policy reported that that as of 
August 2005, more than half of approved SES providers were private, for-profit entities. See 
the Center on Education Policy, From the Capital to the Classroom, Year 4 of the No Child 
Left Behind Act (Washington D.C.; March 2006), for more information. 
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Districts Used Several 
Methods to Contact 
Parents and Offered 
Services on School 
Campuses and at Various 
Times to Increase 
Participation 



Districts have taken multiple actions to encourage participation, as shown 
in table 3. In line with the federal statutory requirement that districts 
notify parents in an understandable format of the availability of SES, over 
90 percent of districts provided written information in English, held 
individual meetings with parents, and encouraged school staff to talk with 
parents about SES. Some districts collaborated with providers to notify 
parents. For example, during our site visit, Illinois state officials described 
a provider and district sharing administrative resources to increase 
participation, which involved the provider printing promotional materials 
and the district addressing and mailing the materials to parents. In 
addition, we estimate that over 70 percent of districts lengthened the 
period of time for parents to turn in SES applications, held informational 
events for parents to learn about providers, and provided written 
information to parents in languages other than English. During our site 
visit to Woodbum, Ore., district officials reported extending the time 
parents had to sign up their children for SES and hosting an event where 
providers presented their programs to parents in English and Spanish. 
Further, Newark, N.J., district officials told us during our site visit that the 
district provided transportation for parents to attend informational events 
to increase participation. Also to encourage participation, an estimated 
90 percent of districts offered services at locations easily accessible to 
students, such as on or near the school campus, and almost 80 percent of 
districts offered services at a variety of times, such as before and after 
school or on weekends. For example, Hamilton County, Tenn., worked 
with providers to offer an early morning tutoring program located at the 
school site in addition to providing services after school. Providers also 
reported delivering SES on school campuses and at various times. 
Specifically, over three-fourths of the 22 providers we interviewed 
reported delivering services at the school site, although providers also 
offered services off-site, such as in the home, online, or at the provider’s 
facility. In addition, providers generally delivered SES after school and 
some also offered SES at alternative times, such as before school, on 
weekends, or during the summer. Finally, about one-third of districts 
provided or arranged for transportation for participating students or 
worked with a local community partner to raise awareness of the services. 
For example, in Newark, N.J., the district worked with a local community 
organization to inform parents and students living in public housing and 
homeless shelters about SES. 
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Table 3: District Actions Taken to Encourage SES Participation (2005-2006) 


Action taken during the 2005-2006 school year 


Estimated percentage 
of districts 


Provided written information in English to parents 


99 


Held individual meetings and/or phone conversations with interested parents 


95 


Encouraged principals, teachers, or other school staff to talk with parents 


93 


Offered supplemental services in locations that are easily accessible to students after school 
(e.g., on or near the school campus) 


90 


Offered SES at a variety of times (e.g., after school, weekends, summer break) 


79 


Lengthened the period of time parents have to submit applications for SES 


79 


Held events where parents of eligible students can learn about providers 


78 


Provided written information in language(s) other than English about SES to parents 


72 


Made public announcements (e.g., television, billboards, newspaper ads, school newsletters) 


67 


Worked with a local community partner to raise awareness of SES (e.g.. Parent Information 
Resource Center) 


39 


Provided or arranged for transportation of students receiving SES to off-site providers 


33 



Source: GAO. 



States also reported taking actions to increase participation in 2005-2006, 
as shown in table 4. Regarding parent notification, all states encouraged 
district staff to communicate with parents about SES. In addition, almost 
90 percent of states provided guidance to districts on the use of school 
campuses for service delivery to encourage participation. 



Table 4: State Actions Taken to Encourage SES Participation (2005-2006) 


Action taken during the 2005-2006 school year 


Percentage of states 


Encouraged district superintendents, supplemental educational services coordinators, or other district staff to 
talk with parents about supplemental educational services 


100 


Increased the number of approved providers 


94 


Provided guidance to districts on the use of school sites to deliver supplemental educational services 


87 


Increased the number of approved providers that offer supplemental educational services online 


73 


Worked with a local community partner to raise awareness of SES (e.g.. Parent Information Resource Center) 


40 


Made public announcements (e.g., television, billboards, newspaper ads) 


23 



Source: GAO. 
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Remaining Challenges 
Include Notifying Parents 
in a Timely and Effective 
Manner and Attracting 
More Providers for Certain 
Areas and Students 



Despite some districts’ promising approaches to encourage participation, 
notifying parents in a timely manner remains a challenge for some 
districts. An estimated 58 percent of districts did not notify parents that 
their children may be eligible to receive SES before the beginning of the 
2005-2006 school year, which may be due in part to delays in states 
reporting which schools were identified for improvement.^ Specifically, 
about half of districts that did not notify parents before the beginning of 
the 2005-2006 school year did not receive notification from the state of the 
schools identified for improvement by that time.^^ Moreover, district 
officials in three of the states we visited experienced delays in receiving 
school improvement information, and state officials agreed that providing 
timely information about whether schools have met state performance 
goals has been a challenge. Almost all of the districts that did not notify 
parents before the beginning of the 2005-2006 school year did so within the 
first 2 months of the year.^'* 



Effectively notifying parents is also a challenge for some districts. For 
example, officials in all four districts we visited reported difficulties 
contacting parents to inform them about SES in part because some 
families frequently move and do not always update their mailing address 
with districts. In addition, some providers we interviewed indicated that 
confusing parental notification letters do not effectively encourage SES 
participation. For example, some of the providers we interviewed said 
some districts use confusing and poorly written letters to inform parents 
of SES or send letters to parents of eligible children but conduct no further 
outreach to encourage participation in SES. Four of the providers we 
interviewed also indicated that complicated district enrollment processes 
can discourage participation. For example, one provider said certain 
districts send parents multiple documents to complete in order for their 
child to receive SES, such as an enrollment form to select an SES provider 
and a separate contract and learning plan. 



GAO previously reported that some states have difficulty notifying schools of their status 
in meeting proficiency goals in a timely fashion in part because of the time involved in 
identifying and correcting errors in student assessment data. See GAO, No Child Left 
Behind Act: Improvements Needed in Education’s Process for Tracking States’ 
Implementation of Key Provisions, GAO-04-734 (Washington D.C.: Sept. 30, 2004), for 
more information. 

This district estimate has a margin of error that exceeds plus or minus 8 percentage 
points. See table 9 in appendix I for more information. 

This district estimate has a margin of error that exceeds plus or minus 8 percentage 
points. See table 9 in appendix I for more information. 
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Another challenge to increasing SES participation is attracting more SES 
providers for certain areas. Some rural districts surveyed indicated that no 
students received services last year because of a lack of providers in the 
area.“ We estimate that the availability of transportation for students 
attending supplemental services was a moderate, great or very great 
challenge for about half of rural districts. For example, one rural district 
commented in our survey that there are no approved providers within 
200 miles of its schools. A few other rural districts commented in our 
sinvey that it was difficult to attract providers to their area because there 
were few students to serve or providers had trouble finding staff to serve 
as tutors. 

In addition, ensuring there are providers to serve students with limited 
English proficiency or disabilities has been a challenge for some districts. 
There were not enough providers to meet the needs of students with 
limited English proficiency in an estimated one-third of districts, and not 
enough providers to meet the needs of students with disabihties in an 
estimated one-quarter of districts. Many states also indicated that the 
number of providers available to serve these groups of students was 
inadequate. While over half of the providers we interviewed reported 
serving students with limited English proficiency or disabilities, some 
providers served these students on a limited basis and reported difficulties 
meeting their needs. For example, one provider reported serving few 
students with limited English proficiency and disabilities because the 
amount of funding the provider receives for SES was not sufficient to pay 
for specialized tutors to serve these students. Another provider said it was 
difficult to find tutors to meet the needs of students with limited English 
proficiency and its program was not designed for students with 
disabilities. Another provider said that it was difficult to serve students 
with disabilities because it required significantly modifying the tutoring 
lessons to meet their needs. 



GAO previously reported that geographic isolation created difficulties for rural districts 
in implementing SES. Specifically, rural district officials stated that traveling long distances 
to meet providers was not a viable option and use of onhne providers was challenging in 
some small rural districts where it was difficult to establish and maintain Internet service. 
See GAO, No Child Left Behind Act: Additional Assistance and Research on Effective 
Strategies Would Help Small Rural Districts, GAO-04-909 (Weishington D.C.: Sept. 23, 
2004), for more details. 

This district estimate has a margin of error that exceeds plus or minus 8 percentage 
points. See table 9 in appendix I for more information. 
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Encouraging student attendance has also been a challenge, in part because 
students may participate in other afterschool activities, such as sports or 
work. Low parent and student demand for SES has been a challenge in 
about two-thirds of districts. For example, about one-quarter of districts 
reported that both competition from other afterschool programs and the 
availability of services that are engaging to students were challenges to 
implementing SES. In addition, providers, district and school officials in all 
four districts we visited told us that SES is competing for students with 
extracurricular and other activities. For example, a Chicago, 111., high 
school official indicated that student attendance at SES sessions declined 
significantly as the school year progressed. To address this problem, 
providers sometimes offer students incentives for participation. For 
example, while 2 of the 22 providers we interviewed offered incentives for 
students to sign up for services, 19 providers used incentives to encourage 
student attendance, such as school supplies and gift certificates. 



Providers Have Taken 
Steps to Deliver 
Quality Services, but 
Contracting and 
Coordination Remain 
Challenges to 
Working with 
Districts and Schools 



To promote improved student academic achievement, providers aligned 
their curriculum with district instruction primarily by hiring district 
teachers and communicating with the teachers of participating students. 
Providers reported communicating with teachers and parents in person as 
well as mailing information and progress reports to them; however, 
districts indicated the extent of provider efforts varied, as some did not 
contact teachers and parents in 2004-2005. In addition, both providers and 
districts experienced contracting and coordination difficulties. In part 
because SES is often delivered in school facilities, providers and officials 
in the districts and schools we visited reported that involvement of school 
administrators and teachers can improve SES delivery and coordination. 



Providers Worked to Align 
Curriculum and 
Communicate with 
Parents, though the Extent 
of Their Efforts Varied 



In order to increase student academic achievement, providers took steps 
to align their curriculum with school instruction by hiring and 
communicating with teachers, though the extent of their efforts varied. A 
majority of the 22 providers we interviewed hired certified teachers in the 
district as tutors. Some providers said hiring district teachers promoted 
curriculum alignment, in part because district teachers were apt to draw 
on district curriculum during tutoring sessions. School officials in three of 
our site visits also said providers’ use of district teachers as tutors helped 
to align the tutoring program with what the student learned in the 
classroom. In addition, some providers reported aligning curriculum by 
communicating with the teachers of participating students to identify 
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student needs and discuss progress. The frequency of contact between 
tutors and teachers ranged from mailing teachers information once prior 
to the beginning of the program to contacting teachers at least weekly, 
according to the providers we interviewed. A few providers also used 
other methods to align their curriculum with district instruction, such as 
using the same tests to evaluate student progress and allowing principals 
to choose components of the tutoring curriculum for students receiving 
SES in their schools. However, not all providers worked with schools to 
align curriculum, as we estimate that some, most, or all providers did not 
contact teachers to align curriculum with school instruction in almost 40 
percent of districts in 2004-2005. For example, Woodburn, Ore., district 
and school officials indicated during our site visit that instead of aligning 
their services with the district curriculum, certain providers openly 
questioned the district’s curriculum and teaching methods, which caused 
confusion among some parents and students. 

Providers reported mailing information as well as meeting with parents 
over the phone and in-person to communicate about student needs and 
progress; however, the frequency of communication with parents varied 
by provider. A majority of the providers we interviewed communicated 
with parents about student progress repeatedly, sometimes by sending 
home progress reports on a monthly basis or holding parent-tutor 
conferences. The frequency of contact between tutors and parents 
reported by the 22 providers we interviewed ranged from meeting with 
parents twice during the tutoring program to giving parents a weekly 
progress report. A few providers also reported communicating with 
parents by holding workshops for parents to learn about the SES program 
and in some cases having the parents sign their students’ learning plans. 
For example, one provider conducted workshops where parents received 
reading materials to share with their children and a parent guide in English 
and Spanish that explained the program and how to use the materials to 
enhance student learning. Some providers also reported hiring staff 
dedicated in part to maintaining communication with parents. However, 
some providers faced difficulties when communicating with parents about 
SES, such as language barriers or incorrect contact information. Districts 
confirmed that the degree to which providers communicated with parents 
varied, as we estimate that some, most, or all providers did not contact 
parents to discuss student needs and progress in about 30 percent of 
districts in 2004-2005. 

Despite these challenges, most districts had positive relationships with 
providers. Specifically, an estimated 90 percent of districts indicated that 
their working relationships with providers during 2004-2005 were good. 
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very good, or excellent. In addition, many of the providers we interviewed 
during our site visits also reported having positive working relationships 
with district officials. Although other studies have found that districts 
reported certain difficulties working with providers, relatively few districts 
reported that their providers signed up ineligible students or billed for 
services not performed in 2004-2005, as shown in figure 3. 



Figure 3: Estimated Percentage of Districts Experiencing Certain Issues with None, 
Few, Some, Most, or All of Their Providers in 2004-2005 
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Source: GAO. 



None/few providers 
Some/most/all providers 
Don’t know 



Generally, states did not hear about these provider issues very often. 
Almost half of states said the issue of providers not showing up for SES 
sessions was rarely brought to their attention.^® Similarly, half of states 
said the issue of providers bilhng the district for services not performed 
was rarely brought to their attention. In addition, about 40 percent of 
states said the issue of providers using excessive incentives was rarely 
brought to their attention. Further, about 40 percent of states said the 



The other states included five that did not respond to the applicable survey question. A 
similar number of states did not respond to the other survey questions discussed in this 
paragraph. 
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issue of providers signing up ineligible students rarely arose. Almost 
one-third of states heard about each of these issues sometimes, while few 
states had these issues brought to their attention very often. For example, 
during our site visits, state officials provided examples of issues that had 
been brought to their attention regarding provider practices, but these 
issues were often isolated incidents particular to one or a few providers in 
certain districts. 



Providers and Districts 
Experienced Contracting 
and Coordination 
Difficulties 



While providers have taken steps to deliver quality services, both 
providers and districts reported experiencing difficulties during the 
contracting process. For example, some of the providers we interviewed 
said certain districts imposed burdensome contract requirements, such as 
requiring substantial documentation to be submitted with invoices, 
limiting the marketing they could do to parents and students, or restricting 
the use of school facilities to deliver services. Specifically, 7 of the 
22 providers we interviewed experienced difficulties with districts 
restricting provider access to school facilities, by for example, not 
allowing providers to deliver services in school buildings or by charging 
providers substantial fees to do so. A few providers also said contracting 
with districts was a resource-intensive process, in part because contract 
requirements vary by district and state. Some of the multi-state providers 
we interviewed reported dedicating a team of staff to help them finalize 
and manage contracts with districts. These providers commented that, 
while they have the administrative capacity to manage this process, 
smaller providers may not have such capacity. In addition, one provider 
that delivered services exclusively online commented that contracting 
with districts across the country was a challenge, particularly because 
some states and districts require provider representatives to attend 
meetings in-person and be fingerprinted in their states. 



Contracting with providers was also a challenge for some districts. 
Specifically, negotiating contracts with providers was a moderate, great, or 
very great challenge in about 40 percent of districts nationwide. For 
example, Woodburn, Ore., district officials described having contractual 
discussions with providers around whether the district would charge fees 
for the use of school facilities, the types of incentives providers could use 
to encourage students to sign up, and whether the district would pay for 
services when students did not attend SES sessions. While states may 
review and define program design parameters as part of the provider 
approval process, district officials in three of our site visits expressed 
concern about their lack of authority to set parameters in provider 
contracts around costs and program design, such as tutor-to-student ratios 
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and total hours of instruction. For example, during our site visit, a 
Hamilton County, Tenn., district official expressed frustration with 
providers that charged the maximum per-pupil amount but varied in the 
level of services provided, such as the number of instructional hours and 
tutor-to-student ratio. Chicago, III, district officials also expressed 
concern about the variation among providers in the hours of instruction 
and cost of services because the district does not have sufficient funds to 
serve all eligible students and officials would like to maximize the number 
of students they can serve. In part to help address district concerns, in 
2005-2006, Illinois required approved providers to submit information on 
the cost of providing services in each of the districts they served and made 
this information available to districts and the public in order to improve 
transparency and accountability. While Tennessee state officials told us 
they were hesitant to set restrictions on providers and would like more 
federal direction on this issue, other states have set restrictions on the 
cost and design of SES programs. For example, Georgia set a maximum 
tutor-to-student ratio of 1:8 for non-computer based instruction and 
1:10 for computer based instruction, and New Mexico adopted a sliding fee 
scale based on the educational level of tutors. 

Coordination of service delivery has also been a challenge for providers, 
districts, and schools. About 70 percent of states reported that the level of 
coordination between providers, districts, and schools implementing SES 
was a moderate to very great challenge. Sometimes these coordination 
difficulties have resulted in service delays. For example, services were 
delayed or withdrawn in three of the districts we visited because not 
enough students signed up to meet the providers’ enrollment targets and 
districts were not aware of these targets. “ In one district we visited, 
services were delayed because school teachers hired to be tutors did not 
provide evidence of their background checks and teaching certificates to 
providers in a timely manner. Coordination difficulties also occurred 
during the enrollment process. Though districts are responsible for 
arranging SES for eligible students, in two districts we visited, both the 
district and providers sent parents enrollment forms, which caused 
confusion among parents as well as additional work for the district staff 
processing the forms. In addition, a few providers told us they do not 



In addition to our analysis, the Center on Education Policy case studies also found that in 
some cases, approved providers that initially expressed interest in serving a certain district 
later decided not to provide services because too few students enrolled. See the Center on 
Education Policy, From the Capital to the Classroom, Year 4 of the No Child Left Behind 
Act (Washington D.C.: March 2006), for more information. 
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know how many students they will serve until enrollment forms are 
returned to district officials, which hinders planning and may delay 
services since they do not know how many tutors they will need to hire 
and train to deliver SES in each district. 



Providers, District and 
School Officials Reported 
That a Greater Role for 
Schools Would Improve 
Local SES Implementation 



In part because SES can be delivered in school facilities, providers and 
officials in the districts and schools we visited reported that involvement 
of school administrators and other staff improves SES implementation. 
Although schools do not have federally-defined responsibilities for 
administering SES, many officials said SES implementation is hindered 
when school officials are not involved. Some providers we interviewed 
said that a lack of involvement of school principals can make it difficult 
for them to coordinate with schools to encourage student participation. In 
addition, a few providers said certain districts contributed to this problem 
by restricting communication with school officials or not defining a role 
for schools in SES implementation. Officials in one of the districts we 
visited also told us that encouraging participation and administering the 
program was more difficult when they did not designate school staff to 
assist the district with SES implementation. School officials from all four 
of our site visits also said the lack of a clear role for school officials, 
including principals, in administering SES has been a challenge. For 
example, Illinois and Oregon school principals told us they found it 
difficult to manage afterschool activities because they didn’t have 
sufficient authority to oversee SES tutors operating in their buildings at 
that time. Further, problems can arise when school officials are not fully 
informed about the SES program. For example, Woodburn, Ore., school 
officials told us that although the school was not provided SES tutoring 
schedules for students, parents and students have come to school officials 
for help when they were unclear about the schedule or when tutors failed 
to show up. 



A majority of the providers we interviewed told us that involvement of 
school principals can improve participation and make it easier to deliver 
services, in part because principals are familiar with the students and 
manage school staff. For example, certain providers reported providing 
principals with information about the tutoring program so that school staff 
can assist with the enrollment process, involving principals in selecting the 
curriculum used in their schools, and sending principals student progress 
reports. In addition, all four districts we visited had school site 
coordinators to assist with SES, such as helping with the enrollment 
process and assisting with the day-to-day administration of the SES 
program in the schools. For example, Woodburn, Ore., district officials 
told us implementation improved when the district designated school site 
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coordinators to assist with parental notification and events where 
providers present their programs, and meet with parents and providers to 
help them complete individual student learning plans. A few providers we 
interviewed also assigned a staff person at the school site to communicate 
with teachers and parents. While helping to administer the SES program 
adds additional administrative burden on schools, school officials in all 
four of the districts we visited said they welcomed a stronger or more 
clearly defined role. 



States’ SES 
Monitoring Has Been 
Limited Though Their 
Efforts Are 
Increasing, and Many 
States Struggle to 
Develop Meaningful 
Evaluations 



While state monitoring of SES had been limited, more states reported 
taking steps to monitor both district and provider efforts to implement 
SES in 2005-2006. In addition, districts have taken a direct role in 
monitoring providers, and their monitoring activities similarly increased 
during this time. Although states are required to withdraw approval from 
providers that fail to increase student academic achievement for 2 years, 
many states reported challenges evaluating SES providers. In addition, the 
few states that have completed an evaluation have not yet produced 
reports that provided a conclusive assessment of SES providers’ effect on 
student academic achievement. 



More States Reported 
Conducting On-Site 
Reviews of Districts in 
2005-2006, and Many also 
Collected Information 
from Other Sources to 
Monitor District SES 
Implementation 



State monitoring of district SES implementation, which is sometimes 
performed as part of state Title I monitoring, had been limited prior to 
2005-2006, though more states reported conducting on-site reviews of 
districts in that year. While less than one-third of states conducted on-site 
reviews of districts to monitor their implementation of SES in 2004-2005, 
almost three-fourths reported conducting such reviews in 2005-2006. This 
increase reflects both those states that had already begun monitoring 
district SES implementation for 2005-2006 at the time of our survey and 
those states planning to conduct monitoring activities before the end of 
that school year. Because our data were collected during the middle of the 
2005-2006 school year, we do not know whether all of the states that 
plarmed to complete these activities before the end of the year did so. In 
both years, a majority of the states that conducted on-site reviews visited 
few or some of their districts. Therefore, while more states reported 
conducting such reviews in 2005-2006 than in 2004-2005, the number of 
districts per state receiving reviews does not appear to have increased. 



In addition to on-site reviews, many states also reported reviewing 
information collected from several other sources to assess district SES 
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implementation in 2005-2006. The most common source used by states 
was district officials, as almost all states reported reviewing or planning to 
review information collected from district officials to monitor their 
implementation of SES in 2005-2006. Further, many states were also 
collecting information from school principals, parents, and providers to 
monitor districts, with between 67 and 81 percent of states reviewing or 
plarming to review information collected from these sources in 2005-2006. 

States also reported reviewing or planning to review information related to 
several aspects of district SES implementation in 2005-2006. For example, 
almost all states reported reviewing district notification of parents and 
SES expenditures, as shown in figure 4. 



Figure 4: Percentage of States That Reviewed Specific Program Eiements to 
Monitor District SES Impiementation in 2005-2006 



Percentage of states 




Program element 



Planned to review before end of 2005-2006 
Reviewed for 2005-2006 at time of survey 



Source: GAO. 

Note: The percentage of states that did not review or plan to review these program elements to 
monitor district SES implementation in 2005-2006 and the percentage of states that did not answer 
these survey questions are not shown in this figure. 



Page 29 



GAO-06-758 No Child Left Behind Act 




States we visited reported that some districts have had difficulties 
implementing SES, in part because of district staff capacity limitations and 
the complexities of administering SES at the local level. When states find 
that a district is having difficulty implementing SES, most hold a meeting 
with the district and provide or arrange for assistance, including 
consultations or training. Half of the states also develop an action plan and 
time line with the district to help improve its efforts. During our site visits, 
state officials reported that notifying parents, maintaining a fair and 
competitive environment for providers, ensuring providers understand 
their SES roles and responsibilities, and determining an appropriate role 
for schools continue to challenge some districts as they implement SES. 



Though States and 
Districts Are Challenged to 
Monitor Providers, They 
Are Moving from Limited 
Monitoring of Providers to 
a More Active Approach 



Although states and districts reported increasing their efforts to monitor 
SES providers between 2004-2005 and 2005-2006, over two-thirds of states 
reported that on-site monitoring of providers has been a challenge. In 
addition, several districts commented in our survey that more provider 
monitoring is needed. During all four of our site visits, state and district 
officials expressed concerns about their capacity to fully administer and 
oversee all required aspects of SES implementation, including provider 
monitoring. Officials explained that state and district capacity to 
implement SES is limited, because there is typically one staff person at 
each level coordinating all of SES, and sometimes that person may also 
oversee implementation of additional federal education programs. Further, 
several states commented in our survey that additional training, technical 
assistance, and national monitoring protocols from the federal government 
would assist their efforts to monitor providers. 



During our site visits, state officials noted that while they did not initially 
have structured plans or procedures in place to monitor SES providers, 
they took steps to develop and formalize procedures starting with the 
2004-2005 and 2005-2006 school years. Nationally, in 2004-2005, states 
monitored providers primarily by collecting data from district officials, 
though many states reported using a more active monitoring approach in 
the next year. For example, while less than one-third of states conducted 
on-site reviews of providers in 2004-2005, over three-fourths had 
conducted or planned to conduct such reviews in 2005-2006, as shown in 
figure 5. In addition, while one-third or fewer states reviewed information 
collected from school staff, parents, and students in 2004-2005, the 
percentage that reported reviewing or planning to review information 
collected from these sources more than doubled the next year. Similar to 
2004-2005, many states continued to rely on information collected from 
district officials to monitor providers in 2005-2006, with almost all states 
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reviewing or planning to review information collected from districts in 
that year. 



Figure 5: Percentage of States Reviewing information Collected from Different 
Sources to Monitor SES Providers in 2004-2005 and 2005-2006 



Percentage of states 
100 

92 




District officiais Parents 
Source of collected information 



On-site reviews 



Schooi staff 



Students 




Planned to review before end of 2005-2006 

Reviewed for 2004-2005 and 2005-2006 (at time of survey) 



Source: GAO. 

Note: The percentage of states that did not review or plan to review information collected from these 
sources to monitor providers in each year and the percentage of states that did not answer these 
survey questions are not shown in this figure. 



Federal guidance suggests states may request district assistance in 
collecting data from providers to assist state monitoring activities. While 
the state is ultimately responsible for monitoring providers, most states 
reported that districts have taken a direct role in monitoring providers. 
Similar to states, although district monitoring of providers was limited in 
2004-2005, districts used a more extensive and active approach in the next 



Page 31 



GAO-06-758 No ChUd Left Behind Act 



year, as shown in figure 6.^° For example, while we estimate that less than 
half of districts collected information from on-site reviews, school staff, 
parents, and students to monitor providers in 2004-2005, 70 percent or 
more were collecting or planning to collect information from these 
sources in 2005-2006. 



Similar to states, districts reported on monitoring activities through our survey in early 
2006; therefore, while some had already begun to monitor providers, others reported on 
their plans to monitor providers before the end of the 2005-2006 school year. The increases 
in monitoring reflect both those districts that had already begun monitoring activities and 
those planning monitoring activities for later in 2006-2006. Because our data were collected 
during the middle of the 2005-2006 school year, we do not know whether the estimated 
percentage of districts that planned to complete these activities before the end of the year 
did so. 
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Figure 6: Estimated Percentage of Districts Reviewing Information Coiiected from 
Different Sources to Monitor SES Providers in 2004-2005 and 2005-2006 

Percentage of districts 
100 

90 86 




School staff Parents On-site reviews Students 

Source of collected information 



I I Planned to review before end of 2005-2006 

I I Reviewed for 2004-2005 and 2005-2006 (at time of survey) 

Source: GAO. 



Although states and districts collected information from similar sources to 
monitor providers, districts collected information from more providers 
than states. Specifically, while a majority of the states that conducted on- 
site reviews observed only some or few providers, we estimate that a 
majority of districts that conducted on-site reviews observed most or all of 
their providers in 2004-2005.^^ While states and districts may both have 
capacity limitations that affect their ability to conduct on-site reviews to 
monitor providers, conducting such reviews is likely easier for districts 
because services are often delivered on or near school campuses. 



The district estimate has a margin of error that exceeds plus or minus 8 percentage 
points. See table 9 in appendix I for more information. 
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States and districts collected information on several aspects of SES 
programs to monitor providers, as shown in table 5. While federal 
regulations provide states flexibility to design their own SES monitoring 
systems, over two-thirds or more of states and districts monitored or 
planned to monitor all program elements listed, including those related to 
service delivery and use of funds. For example, 94 percent of states 
monitored or planned to monitor parent or student satisfaction with 
providers, and 93 percent of districts monitored or planned to monitor 
billing and payment for services and student attendance records. 



Table 5: Percentage of States and Districts That Reviewed Specified Program Eiements to Monitor Providers in 2005-2006 



Percentage of states Estimated percentage of districts 

Monitored Monitored 

Pianned to or planned Planned to or planned 



Program element 


Monitored 


monitor 


to monitor 


Monitored 


monitor 


to monitor 


Parent/student satisfaction with a provider 


27 


67 


94 


34 


57 


91 


Provider communication with teachers and 
parents 


37 


56 


92 


46 


43 


89 


Extent to which a provider’s program, as 
enacted, reflects its program design, as outlined 
in its application to your state 


19 


73 


92 


30 


41 


70 


Evidence of meeting academic achievement 
goals as stated on student learning plan 


23 


65 


88 


28 


60 


88 


Evidence of improved student achievement 
based on any statewide assessment 


15 


71 


87 


26 


65 


91 


Alignment of provider curriculum with 
district/schooi curriculum or instruction 


25 


62 


87 


35 


39 


74 


Student attendance records 


27 


56 


83 


67 


25 


93 


Evidence of improved student achievement 
based on provider assessments 


27 


56 


83 


39 


52 


91 


Protection of student privacy 


33 


50 


83 


55 


28 


82 


Adherence to applicable health, safety, and civil 
rights laws 


29 


48 


77 


48 


26 


74 


Provider financial stability (e.g., audits, financial 
statements) 


31 


42 


73 


N/A 


N/A 


N/A 


Evidence of improved student achievement 
based on grades, promotion, and/or graduation 


12 


58 


69 


23 


57 


80 


Billing and payment for services 


N/A 


N/A 


N/A 


72 


21 


93 



Source: GAO. 

Note: The percentage of states that did not review or plan to review these program elements to 
monitor providers in 2005-2006 and the percentage of states that did not answer these survey 
questions are not shown in this table. In addition, we did not ask states if they monitored billing and 
payment for services, and we did not ask districts if they monitored provider financial stability. 



Page 34 



GAO-06-758 No Child Left Behind Act 







Many States Struggle to 
Develop Meaningful 
Evaluations, and the Few 
State Evaluations 
Completed to Date Were 
Inconclusive 



Many states struggle to develop evaluations to determine whether SES 
providers are improving student achievement, though states are required 
to evaluate and withdraw approval from providers that fail to do so after 
2 years. Specifically, federal law requires states to develop standards and 
techniques to evaluate the services delivered by approved providers, but it 
does not require states to use specific evaluation methods or criteria for 
determining provider effectiveness.^^ Through our survey, states reported 
several challenges to evaluating SES providers. Specifically, over 
three-fourths of states reported that determining sufficient academic 
progress of students, having the time and knowledge to analyze SES data, 
and developing data systems to track SES information have been 
challenges. Further, during our site visits to Illinois and New Jersey, state 
officials noted they were currently in the process of improving their data 
collection systems to more effectively capture and analyze data for SES 
evaluations. In addition, several state officials reported that while they 
have collected some information to assess provider effectiveness, they 
have done little with that data. Others noted that they have not received 
sufficient federal guidance on effective models for SES provider 
evaluations, and because developing and implementing such evaluations 
can be both time-consuming and costly, additional assistance from 
Education would improve their efforts. 



At the time of our survey in early 2006, only a few states had drafted or 
completed an evaluation report addressing individual SES provider’s 
effects on student academic achievement, and we found that no state had 
produced a report that provided a conclusive assessment of this effect. 
New Mexico and Tennessee were the only two states that had completed 
final or draft SES evaluation reports that attempted to assess the impact of 
all SES providers serving students in their states in previous years.^^ To 
measure student academic achievement. New Mexico’s report analyzed 
students’ grades as well as their scores on state assessments and provider 
assessments, which often d iff er by provider and are administered both 
before SES sessions begin and at the end of SES sessions each year. 



Further, the federal SES guidance suggests that student performance can be measured in 
a variety of ways, including with provider assessments or state or district standardized 
assessments. However, the guidance does not discuss the limitations of using different 
approaches to conduct such evaluations. 

At the time of our survey, several additional states, including Louisiana and Pennsylvania, 
were in the process of drafting an SES evaluation report that would assess the impact of 
SES providers serving students in their states in previous years, but the reports were not 
yet available to the public. 
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However, the report noted that these assessments produced d iff erent 
results related to student academic achievement gains. While Tennessee 
also planned to review students’ state assessment scores, the draft 
available at the time of our analysis did not include this information. In 
addition, both reports drew on information obtained through other 
sources, such as teacher surveys, to assess provider effectiveness. Due to 
their limitations, neither evaluation provided a conclusive assessment of 
SES providers’ effect on student academic achievement. 

In addition, at the time of our survey, over half of the states reported that 
they were in the process of conducting an evaluation of SES providers in 
order to meet the federal requirement of assessing each provider’s effect 
on student academic achievement.^'* Similar to the state evaluations 
already undertaken, officials reported using d iff erent methods and criteria 
to evaluate SES providers. For example, some states were collecting each 
provider’s pre- and post-SES assessments of students while others were 
collecting student achievement data from districts or students’ state 
assessment scores. Further, while one state defined adequate student 
progress as 80 percent of a provider’s students making one-grade level of 
improvement after a year of SES, another state defined adequate student 
progress as 50 percent or more of a provider’s students having any positive 
academic achievement gain after a year of SES. While these states have yet 
to produce final results from their SES provider evaluations, it is unclear 
whether any of these efforts will produce a conclusive assessment of SES 
providers’ effect on student academic achievement. 

Likely because states are struggling to complete evaluations to assess SES 
providers’ effect on student academic achievement, states did not report 
that they have withdrawn approval from providers because their programs 
were determined to be ineffective at achieving this goal.**** Rather, though 
over 40 percent of states reported that they had withdrawn approval from 
some providers, they most frequently reported withdrawing provider 
approval because the provider was a school or district that had entered 



In addition, at least two school districts, Chicago and Minneapolis, have taken steps to 
evaluate SES, but we found that their evaluation efforts have also not yet produced a 
conclusive assessment of SES’s impact on student academic achievement. 

Only one state reported withdrawing approval from one of its providers because that 
provider’s program was generally ineffective. However, this provider’s program was found 
to be ineffective because the provider did not deliver services to all of the students it 
enrolled. This state also indicated that it had not yet completed an evaluation of SES’s 
effect on student academic achievement. 
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needs improvement status, the provider asked to be removed from the 
state-approved provider list, or because of provider financial impropriety. 



Several Education 
Offices Monitor and 
Support SES 
Implementation, but 
States and Districts 
Cite the Need for 
Additional Assistance 
and Flexibility 



Several offices within Education monitor various aspects of SES activity 
across the country and provide support, but states and districts reported 
needing additional assistance and flexibility with program implementation. 
Education conducts SES monitoring in part through policy oversight and 
compliance reviews of states and districts, and provides SES support 
through guidance, grants, research, and technical assistance. However, 
many states and districts reported needing additional assistance and 
guidance regarding evaluation and administration of SES. Further, some 
states and districts voiced interest in expansion of Education’s pilot 
programs that increase SES flexibility, including the pilot that allows 
certain low-achieving districts to serve as SES providers. 



Education Monitors SES Oil and OESE are primarily responsible for monitoring and supporting 

and Provides States and state and district SES implementation, and other Education offices also 

Districts with Guidance contribute to these efforts (see fig. 7). 

and Technical Assistance 
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Figure 7: U.S. Department of Education Offices Monitoring and Supporting SES 




Monitoring 



Performs “desk monitoring” 
of SES information 
Performs ad-hoc monitoring 
of SES issues as they arise 



Monitoring 



Performs on-site visits to 
states and districts as part 
of Titie i monitoring 
Oversees coiiection and 
anaiysis of Consoiidated 
State Performance Reports, 
inciuding SES data 



Monitoring 



Performed six state SES 
audits during 2003-2004 
and 2004-2005 
Performed audits of SES 
impiementation and 
providers in five CA 
districts during 2005 



Support 



Support 



Support 



Support 



• Ciarifies SES poiicy and 
provides technicai assistance, 
in part through presentations 

• Coordinates the pubiication of 
non-reguiatory SES guidance 

• issued grant for SES 
technicai assistance 
provided by SESQC 

• Issued grants to increase 
minority participation in SES 

• Issued grants to increase 
SES access in rural areas 



Clarifies SES policy and 
provides technical assistance, 
in part through presentations 
Issued grant to fund 
Comprehensive Center on 
Innovation and 
Improvement that provides 
technical assistance for 
Education programs, including 
SES 



Issued several reports focused 
on Title I, including SES (e.g. 
National Assessment of Title I: 
Interim Report) 

Produced a nine-district case 
study report, Early 
Implementation of SES under 
the NCLBA, with a follow-up 
report for 2003-2004 



Provides technical 
assistance to faith- and 
community-based 
organizations interested in 
becoming SES providers 



Source: GAO analysis. 

Note: This figure reflects the coordination of Education's offices rather than the statutory reporting 
relationship. 



on, which leads SES policy development and provides strategic direction, 
monitors SES policy issues primarily through what it calls “desk 
monitoring.” This monitoring is performed at its federal office and 
includes the review of SES-related research and media reports. Oil also 
conducts more intensive monitoring of specific SES implementation 
challenges when states, districts, and providers bring them to Education’s 
attention and keeps a log documenting these issues. For example, during 
2004-2005, Illinois and New Jersey officials contacted Oil to clarify 
guidance regarding providers affiliated with districts in need of 
improvement, and Oil staff provided assistance and clarification. In 
addition, several providers we interviewed also mentioned that they have 
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contacted Oil directly to discuss implementation challenges associated 
with enrollment, district contracts, and provider access to school facilities. 



OESE, which oversees and supports NCLBA implementation, is also 
involved in monitoring SES implementation through its overall monitoring 
of state compliance with Title I and NCLBA. To monitor Title I, OESE staff 
visit state departments of education and selected districts within each 
state to interview officials and review relevant documents. Once the 
review is complete, OESE issues a report to the state outlining any 
instances of Title I non-compliance, including those related to SES, and 
actions needed to comply with regulations. As of June 2006, OESE had 
visited and issued reports to over three-fourths of the states. 

In addition to its Title I monitoring, OESE also oversees the collection of 
state NCLBA data, including data on SES, through the annual Consolidated 
State Performance Report (CSPR). For the CSPR, each state is required to 
report the number of schools with students receiving SES, the number of 
students eligible for services, and the number that received services.^® 
However, although almost all states reported that they are collecting 
information on district SES expenditures as part of their oversight. 
Education does not require states to submit information on the percent or 
amount of Title I funds districts spent for SES through the CSPR or other 
means. Therefore, while Education tracks the extent to which states are 
providing SES to eligible students, the department does not collect data on 
the relative costs of serving them. Further, under NCLBA, Education is 
required to present an annual summary of the CSPR data to Congress. As 
of June 2006, the most recent report to Congress was for the 2002-2003 
school year, though Education officials indicated they expect to submit 
reports for 2003-2004 and 2004-2005 in the near future. 

While on and OESE monitoring of SES has been either limited to desk 
monitoring or combined with general Title I monitoring, OIG has 
conducted audits specifically focused on SES. During 2003-2004 and 2004- 
2005, the office performed a series of SES implementation audits in six 
states, which involved site visits to state offices and selected districts 
within each state.^^ Also during 2004-2005, OIG performed audits of five 



States have only reported the number of students eligible for SES since the 2003-2004 
CSPR. Also, for the 2004-2005 CSPR, Education gave states the option to also report the 
number of students who applied for SES. 

States audited were Delaware, Illinois, Indiana, Michigan, New Jersey, and Nevada. 
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California districts and one SES provider within each district. These 
audits included an examination of district SES contracts with providers, 
provider services, and provider compliance with state SES regulations. 

Several Education officials reported coordinating internally to share 
information, including SES monitoring results. To facilitate coordination. 
Oil leads an internal group comprised of staff members from other 
Education offices, who meet bi-weekly to exchange information. OESE 
shares its state Title 1 monitoring results and CSPR data with other 
Education offices. In addition, OIG makes recommendations to both Oil 
and OESE in its state and district SES auditing reports and disseminates 
the reports throughout Education and on the department’s Web site. Since 
2002, Oil has coordinated the publication of four versions of 
non-regulatory SES guidance, each updated to address ongoing challenges 
with SES implementation. The latest and most comprehensive version of 
non-regulatory SES guidance was published in June 2005. In May 2006, 
Education issued a separate supplement to the guidance containing 
additional information on private school participation in providing SES 
and a policy letter clarifying the definition of a district-affiliated provider. 

In addition to its monitoring efforts. Oil also provides SES implementation 
assistance, in part through presentations at conferences, and through 
grants to external organizations that assist states and districts. For 
example. Oil staff have presented information on SES policy and 
promising practices at national meetings attended by SES coordinators 
and others involved in SES implementation. In addition, the office has 
issued grants to the Black Alliance for Educational Options, the Hispanic 
Council for Reform and Educational Options, and through the Star Schools 
Program to promote SES to minority students and those in rural areas. 
Further, Oil funded the Supplemental Educational Services Quality Center 
(SESQC), which offered SES technical assistance through tool-kits, issue 
briefs, and a Web site containing SES information for state and district 
officials, schools, parents, and providers. SESQC also periodically 
convened representatives of organizations working on education issues to 
discuss SES national coordination, challenges, and promising practices. 
However, those meetings and all SESQC activities were discontinued in 
December 2005 when SESQC’s grant period ended. 



California districts audited were Los Angeles Unified, Oakland Unified, Salinas Union 
High School, San Diego City Schools, and Stockton Unified. 
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Other Education offices also provide SES support through various means. 
For example, OESE funded the Comprehensive Centers Program through 
grants that established technical assistance centers across the country to 
help low-performing schools and districts close achievement gaps and 
meet the goals of NCLBA. Of these, the Center on Innovation and 
Improvement provides support to regional centers that assist states with 
Education’s programs, including SES. In addition. Education’s Policy and 
Program Studies Service, within the Office of Planning, Evaluation and 
Policy Development, oversees several research studies that examine SES, 
either in whole or in part. These reports, such as the National Assessment 
of Title I: Interim Report and Case Studies of Supplemental Services 
under the No Child Left Behind Act, provide states and districts with 
information on SES implementation, challenges, and promising practices. 
Further, Education’s Center for Faith-Based and Community Initiatives 
offers technical assistance to faith- and community-based organizations 
interested in becoming state-approved SES providers. 

Given the technical assistance and support Education has already 
provided to states and districts for implementation of SES and school 
choice, and the department’s view that implementation of these provisions 
has been uneven throughout the country, in May 2006, Education issued a 
policy letter announcing the department’s plans to take significant 
enforcement action. Specifically, Education plans to use the data collected 
through its monitoring and evaluation efforts to take enforcement actions 
such as placing conditions on state Title I grants, withholding federal 
funds, or entering into compliance agreements. In the letter, the 
department noted that its various monitoring activities have identified 
several areas of noncompliance with SES requirements. For example, the 
OIG’s audits found that each of the six states reviewed failed to adequately 
monitor their districts for compliance. Consequently, nearly all of the 
parental notification letters reviewed failed to include the required key 
components, and several districts failed to budget sufficient funding for 
services. Through our own analysis of Education’s monitoring reports, we 
also found that some of the states reviewed were found to have inadequate 
or incomplete processes for monitoring and evaluating SES providers. 



Page 41 



GAO-06-758 No Child Left Behind Act 




Many States and Districts 
Reported Needing SES 
Evaluation Support and 
Additional Technical 
Assistance, and Some 
Suggested Increasing 
Flexibility through 
Education’s Pilot Programs 



Despite Education’s efforts, many states and districts reported needing 
more information and assistance to better comply with certain aspects of 
SES implementation, including SES evaluation (see table 6). Specifically, 
85 percent of states and an estimated 70 percent of districts needed 
additional assistance with methods for evaluating SES, and over 
60 percent also needed assistance with developing data systems. Many 
districts also needed more information on provider quality and 
effectiveness. Although OESE and OIG monitoring results have also 
continually indicated that states and districts struggle with SES evaluation. 
Education has yet to provide comprehensive assistance in this area, and 
during our site visits, officials mentioned that they have been relying on 
other states, organizations, or individuals for evaluation assistance. States 
and districts also indicated a need for more support and technical 
assistance to help them administer SES. Specifically, approximately 
three-fourths of states and two-thirds of districts reported needing funding 
to increase their capacity to implement SES. Many states also reported 
needing tool kits and model/sample documents, as well as training from 
Education, and a majority of districts needed effective parent outreach 
strategies. Further, most states reported a need for corrferences or 
meetings where they could share lessons learned and promising practices 
with other states. A few Termessee officials mentioned that conferences 
hosted by national organizations have been an effective means of allowing 
SES officials to gather and share knowledge. 
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Table 6: Percentage of States and Districts in Need of Different Types of information or Assistance with SES implementation 




Percentage 
of states 




Estimated percentage 
of districts 


Evaluation information or assistance needed 


Methods for evaluating SES 


85 


Information on provider quaiity or 
effectiveness 


78 


Developing data systems 


71 


Methods for evaluating SES 


70 






Developing data systems for 
monitoring and evaluation 


64 


Other support or technical assistance needed 


Conferences/meetings with other 
states to share lessons learned/ 
promising practices 


90 


Funding to increase district capacity to 
implement SES 


67 


Tool kits and model/sample 
documents 


85 


Information on effective parent 
outreach strategies 


61 


Training 


85 


Assistance with contract negotiation 


35 


Funding to increase state capacity to 
implement SES 


77 






Approving and monitoring providers 


71 






Allowances for SES to be offered 
before school choice provisions 


58 






Clarification of roles for states, 
districts, and providers in implementing 
SES 


46 






Waivers for states for exclusion from 
specific SES provisions 


44 






Clarification of the 20-percent set- 
aside requirements and carryover 
allowances 


44 







Source: GAO. 



While three-fourths of states reported that the most recent version of 
Education’s SES guidance, published in June 2005, has been very or 
extremely useful, several states commented through our survey that they 
needed additional or clearer guidance on certain SES provisions. This 
included guidance on managing SES costs and fees, implementing SES in 
rural areas, and handling provider complaints. During three of our site 
visits, officials also expressed some concern about the lack of clarity in 
the SES guidance with regards to student eligibility requirements and how 
to craft a parental SES notification letter that is both complete and easy 
for parents to understand. 
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Regarding parental notification letters, though both OESE and OIG found 
many states and districts to be non-compliant with the federal requirement 
that district SES parental notification letters include several specific 
elements,^® Education’s SES guidance, which is coordinated by Oil, 
provides a sample that does not clearly specify all of the key elements 
required by SES law and regulations. For example, the sample letter does 
not include information on provider services, qualifications, and 
effectiveness. Furthermore, a few state and district officials commented 
that, when followed, the Title I regulations governing SES yield a letter 
that is unreasonably long and complex, which may be difficult for parents 
to understand. 

Many states and districts expressed interest in the flexibility offered 
through two pilot programs that Education implemented during 2006-2006. 
The department designed these pilots to increase the number of eligible 
students receiving SES and to generate additional information about the 
effect of SES on student academic achievement. For example, several 
state and district SES coordinators expressed interest in Education’s pilot 
program that allowed two districts in needs improvement status to act as 
SES providers in exchange for their expansion of student access to SES 
providers and collection of achievement data to determine SES program 
effectiveness. During three of our four site visits, state and district officials 
expressed concern that districts identified for needs improvement are 
excluded from delivering SES, and one state official noted that removing 
districts from the state approved provider list may result in lower SES 
participation. Further, in our surveys, three state SES coordinators and 
17 district SES coordinators wrote in comments that permitting districts in 
needs improvement status to provide services would assist their efforts. 
Through both our surveys and site visits, officials suggested that allowing 
districts to act as providers may ease student access to SES for rural 
districts that do not have providers located nearby, allow more students to 
participate in SES because district costs to provide services are sometimes 
lower than other providers’ costs, and enable districts to continue their 
existing tutoring programs that they feel are effective and meet the same 
goals as SES. Overall, we estimate that 15 percent of districts were state- 
approved providers in 2004-2005. However, another national survey 



As discussed earlier, OIG found all six of the states it visited to be deficient with respect 
to parent notifications. In addition, in our analysis of OESE Title I monitoring reports 
issued as of June 2006, we found that OESE cited 9 of the 40 states it had visited for SES 
non-compliance with respect to district parent notifications. 
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recently found the percentage of urban and suburban districts that are 
state-approved SES providers is declining. 

The other SES pilot allowed four districts in Virginia to offer SES instead 
of school choice in schools that have missed state performance goals for 
2 years and are in their first year of needs improvement. During our site 
visits and through our surveys, many states and districts expressed 
interest in adjusting the order of the SES and school choice interventions. 
Specifically, half of states and over 60 percent of districts suggested that 
SES should be made available before school choice (see table 7). Further, 
approximately three-quarters of both states and districts indicated that 
SES should be offered either before or simultaneously with choice. As we 
found through our previous work on school choice, few students are 
opting to transfer schools in the first year of needs improvement, and 
therefore this change would provide students with another option to 
receive additional academic support in that year. Further, in a recent 
national study, district and school officials noted that parents and students 
are often not interested in changing schools, in part because of potential 
long commutes and satisfaction with their current schools, which suggests 
that parents and students would likely prefer to receive SES in their own 
schools and neighborhoods rather than school choice.^^ In line with 
interest in increased flexibility with these interventions, in May 2006, 
Education announced that due to the positive results in Virginia districts 
under the pilot, the department plans to extend and expand this pilot in 
2006-2007. 



Table 7: State and District Opinion on the Ordering of School Choice and SES 
In percent 

Order of school choice and SES States District 

SES should precede school choice 48 62 

Both school choice and SES should be offered at the same time 27 1 5 

School choice should precede SES 15 23 

Source: GAO. 

Note: 10 percent of states did not respond or were not sure. In addition, district percentages are 
estimates. 



See the Center on Education Policy, From the Capitol to the Classroom: Year 4 of the No 
Child Left Behind Act (Washington D.C.: March 2006). 

Ibid. 
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Conclusions 



Over the last few years, almost all states and approximately 1,000 districts 
have been required to implement SES across the country and, if current 
trends continue, more schools will be required to offer services in the 
future. Although some states and districts are beginning to gain experience 
in implementing SES and use promising approaches to increase SES 
participation, many students are still not receiving services, in part 
because providers are sometimes not available to serve certain areas and 
groups. In addition, some districts are unsure how to involve school 
officials in facilitating local coordination of SES implementation and 
providing effective parental notification. While Education has provided 
support to states and districts through guidance and technical assistance, 
many report needing additional assistance to address these challenges. 
Further, the lack of clarity between policy guidance issued by Oil and 
criteria used by OESE in their compliance reviews of states’ 
implementation efforts creates additional challenges in meeting the federal 
requirements for parental notification letters. Providing states with clear 
guidance that has been coordinated across Education offices is especially 
important now that the department has announced plans to take 
significant enforcement actions to ensure states comply with federal SES 
requirements. 

While some districts do not have any students receiving services and, 
therefore, are not spending any Title I funds for SES, other districts are 
spending more than their entire set-aside and still have students on waiting 
lists to receive services. Two districts have been able to participate in 
Education’s pilot program waiving federal regulations that preclude 
districts in need of improvement from providing SES, which may help 
them provide services to more students at a lower cost. However, 
extending this flexibility to more districts depends on the evaluation of the 
quality of these services to determine if SES is having a positive impact on 
student academic achievement. In addition, the absence of strategies that 
states can use to set parameters around program design and costs further 
hinders their ability to stretch available funding to serve more students. 
Federal and state oversight of district efficiency in using federal funds to 
provide SES services is hindered by incomplete reporting requirements 
that require states to report on the number of eligible children receiving 
SES, but not the data they collect on the amount of Title I funding used to 
serve them. This information gap limits Education’s ability to track state 
and district compliance in spending funds for SES. Further, Education’s 
ability to ensure that federal dollars are effectively spent to improve 
student academic achievement is limited until states increase their 
capacity to monitor and evaluate provider performance. In the absence of 
additional federal technical assistance and access to irrformation about 
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state and district promising practices, some states may continue to 
struggle with implementation and evaluation of SES. 



Recommendations 



To help states and districts implement SES more effectively, we 
recommend that the Secretary of Education use the department’s Web site 
and the Center on Innovation and Improvement, as well as other means of 
communication, to: 

• Provide federal guidance on SES parental notification letters that is 
clear and has been coordinated internally between Oil and OESE to 
provide additional assistance to states and districts to help them 
comply with federal requirements and ensure that letters are easy for 
parents to understand. Education might consider providing several 
samples of actual district notification letters that meet these criteria. 

• Collect and disseminate information on promising practices used by 
states to attract more providers for certain areas and groups and 
promising practices used by districts to improve parental notification 
of SES services and providers’ ability to serve specific groups of 
students and to encourage student attendance. 

• Provide examples of how districts can involve schools and school 
officials to facilitate local coordination with providers. 

To improve states’ and districts’ ability to make the most of funding for 
SES and provide services to the maximum number of students, we 
recommend that the Secretary of Education: 

• Consider expanding the 2005-2006 pilot that allowed two districts in 
need of improvement to enter into flexibility agreements to serve as 
SES providers if evaluation results show that these districts can 
provide quality SES services. 

• Clarify what states can do through the provider approval process to set 
parameters around program design and costs. For example. Education 
could issue guidance to states that clarifies their authority to set 
parameters on issues such as minimum hours of SES per student, 
minimum tutor qualifications, and cost ranges. In addition. Education 
could suggest to states that they coordinate these discussions with 
districts to address their concerns about program design and costs. 

To improve federal and state monitoring of SES, we recommend that the 
Secretary of Education: 

• Require states to report information necessary to determine the 
amount of funds spent by districts to provide SES and the percentage 
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Agency Comments 



of their Title I allocations that this amount represents. Because almost 
all states reported that they are plarming to monitor district SES 
expenditures, Education could require the states to submit these data 
through the annual NCLBA Consolidated State Performance Reports. 

• Provide states with technical assistance and guidance on how to 
evaluate the effect of SES on student academic achievement. For 
example, Education might require the Center on Innovation and 
Improvement to focus its SES assistance on providing states with 
suggested evaluation methods, measures to assess the impact of SES 
on achievement, and criteria for using this information to monitor and 
withdraw state approval from providers. Further, lessons learned and 
promising practices on evaluation could also be shared with states on 
the Center on Innovation and Improvement’s Web site or during 
national or regional meetings, trainings, or conferences. 



We provided a draft of this report to Education for review and comment. 
Educations’ written comments are reprinted in appendix II, and the 
department's technical comments were incorporated into the report as 
appropriate. In its written comments. Education expressed appreciation 
for the report’s recommendations and cited actions the department has 
already initiated or plans to take in addressing them. Specifically, 
Education noted several projects under development that might assist in 
carrying out our recommendations to provide more assistance to states on 
notifying parents, attracting providers for certain areas and groups, and 
involving schools. The department also said that is currently considering 
expanding the pilot allowing districts in need of improvement to apply to 
become SES providers, per our recommendation. Regarding our 
recommendation that Education clarify what states can do to set 
parameters around program design and costs. Education said it would 
consider addressing this issue further in the next set of revisions to the 
SES non-regulatory guidance. In addition. Education said it would address 
our recommendations to improve federal and state monitoring of SES by 
proposing that districts report on their SES spending and by providing 
more SES evaluation assistance to states through an updated issue brief as 
well as technical assistance provided by the Comprehensive Center on 
Innovation and Improvement and at a conference this fall. 



We are sending copies of this report to appropriate congressional 
committees, the Secretary of Education, and other interested parties. 
Copies will also be made available upon request. In addition, the report 
will be available at no charge on GAO’s Web site at http://www.gao.gov. If 
you or your staff have any questions about the report, please contact me at 
(202) 512-7215. Contact points for our Offices of Congressional Relations 
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and Public Affairs may be found on the last page of this report. GAO staff 
who made major contributions to this report are listed in appendix 111. 




Mamie S. Shaul 

Director, Education, Workforce, and 
Income Security Issues 
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Appendix I: Scope and Methodology 



To obtain nationally representative information on supplemental 
educational services (SES) participation, state and local implementation, 
and federal oversight, we conducted a Web-based survey of state SES 
coordinators and a mail survey of district SES coordinators from a 
nationally representative sample of districts with schools required to offer 
SES. We also conducted site visits during which we interviewed state, 
district, and school officials representing four school districts, and we 
conducted interviews with 22 SES providers both during the site visits and 
separately. In addition, we spoke with staff at Education involved in SES 
oversight and implementation and reviewed Education’s data on SES. We 
conducted our work from August 2005 through July 2006 in accordance 
with generally accepted government auditing standards. 



Survey of States To better understand state SES implementation, particularly how states 

are monitoring and evaluating SES, we designed and administered a Web- 
based survey of state SES coordinators in all 50 states, the District of 
Columbia, and Puerto Rico. The survey was conducted between January 
and March 2006 with 100 percent of state SES coordinators responding. 
The survey included questions about student participation in SES, actions 
taken to increase participation, SES funding and expenditures, methods 
used to monitor and evaluate implementation, implementation challenges, 
and assistance received from Education. 

Because this was not a sample survey, there are no sampling errors. 
However, the practical difficulties of conducting any survey may introduce 
nonsampling errors, such as variations in how respondents interpret 
questions and their willingness to offer accurate responses. We took steps 
to minimize nonsampling errors, including pre-testing draft instruments 
and using a Web-based administration system. Specifically, during survey 
development, we pre-tested draft instruments with officials in Oregon, 
Maryland, and Washington between October and November 2005. In the 
pre-tests, we were generally interested in the clarity of the questions and 
the flow and layout of the survey. For example, we wanted to ensure 
definitions used in the surveys were clear and known to the respondents, 
categories provided in closed-ended questions were complete and 
exclusive, and the ordering of survey sections and the questions within 
each section was appropriate. On the basis of the pre-tests, the Web 
instrument underwent some slight revisions. A second step we took to 
minimize nonsampling errors was using a Web-based survey. By allowing 
respondents to enter their responses directly into an electronic 
instrument, this method automatically created a record for each 
respondent in a data file and eliminated the need for and the errors (and 
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costs) associated with a manual data entry process. To further minimize 
errors, programs used to analyze the survey data and make estimations 
were independently verified to ensure the accuracy of this work. 

While we did not fully validate specific information that states reported 
through our survey, we took several steps to ensure that the information 
was sufficiently reliable for the purposes of this report. For example, after 
the survey was closed, we made comparisons between select items from 
our survey data and other national-level data sets.' We found our survey 
data were reasonably consistent with the other data sets. On the basis of 
our checks, we believe our survey data are sufficient for the purposes of 
our work. 



Survey of School Districts To obtain national-level information on district implementation of SES, we 

administered a mail survey to a nationally representative sample of 
districts that had schools required to offer SES in school year 2004-2005. 
The survey was conducted between January and March 2006. To obtain 
the maximum number of responses to our survey, we sent a reminder 
postcard to nonrespondents approximately 1 week after the initial mailing 
of the survey instrument, a follow-up mailing with the full survey 
instrument to nonrespondents approximately 3 weeks after the initial 
mailing, and a second follow-up mailing with the full survey instrument 
approximately 4 weeks later. The survey included questions about student 
participation in SES, characteristics of students receiving SES, actions 
taken to increase participation, SES funding and expenditures, methods 
used to monitor and evaluate implementation, implementation challenges, 
and assistance received and still needed. 

Population The target population of 1,095 districts consisted of public school districts 

with at least one school in each of their jurisdictions required to provide 
SES in the 2004-2005 school year. To define our population, we collected 
school improvement information from state education agency Web sites 
and the NCLBA Consolidated State Performance Reports: Part 1 for 



' We compared our state survey data to data on SES reported by Education in the 
Evaluation of Title I Accountability Systems and School Improvement Efforts: Findings 
From 2002-03 (Washington, D.C.: Nov. 2005) and the National Assessment of Title I: 
Interim Report (Washington, D.C.: Feb. 2006). We also compared our survey data to data 
on SES reported by the Center on Education Policy in From the Capital to the Classroom: 
Year 3 of the No Child Left Behind Act (Washington, D.C.: March 2005) and From the 
Capital to the Classroom: Year 4 of the No Child Left Behind Act (Washington, D.C.: 
March 2006). 
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Sample Design and Errors 



Reporting on School Year 2003-2004 that states submitted to Education. 
When available, we checked both sources for school improvement 
information and used the source that provided the most recently updated 
data, as this data is often updated by states over the course of the school 
year. After constructing our population of districts, we used Education’s 
Common Core of Data Local Education Agency (School District) 
preliminary file and the Public Elementary/Secondary School preliminary 
file for the 2003-2004 school year to further define the characteristics of 
our population. On the basis of our review of these data, we determined 
these sources to be adequate for the purposes of our work. 

The sample design for the mail survey was a stratified random sample of 
districts with one certainty stratum containing all of the districts with 
100,000 students or more and one stratum containing all other districts in 
the universe. We included the 21 districts with 100,000 or more students 
with certainty in the sample to ensure we gathered information from the 
largest districts required to offer SES. 

We selected a simple random sample of districts in the non-certainty 
stratum and calculated the sample size to achieve a precision of plus and 
minus 7 percent at the 95 percent corrfidence level for an expected 
proportion of 60 percent. To ensure the sample sizes were adequate, we 
increased the sample size assuming we would obtain a 70 percent 
response rate. The total sample size for this stratum was 237 districts. In 
the sample, each district in the population had a known, nonzero 
probability of being selected. Each selected district was subsequently 
weighted in the analysis to account statistically for all the schools in the 
population, including those that were not selected. Table 8 provides a 
description of the universe and sample of districts. 



Table 8: Description of the Population and Sample of Districts 


Stratum 


Population 


Sample size 


District enroilment >= 100,000 


21 


21 


District enroilment < 100,000 


1,074 


237 


Total 


1,095 


258 



Source: GAO. 



Because we surveyed a sample of districts, our results are estimates of a 
population of districts and thus are subject to sampling errors that are 
associated with samples of this size and type. Our confidence in the 
precisions of the results from this sample is expressed in 95 percent 
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confidence intervals, which are expected to include the actual results in 
95 percent of the samples of this type. We calculated confidence intervals 
for this sample based on methods that are appropriate for a stratified 
random sample. 

We determined that 10 of the sampled districts were out of scope because 
they did not have any schools required to provide SES in the 2004-2005 
school year. All estimates produced from the sample and presented in this 
report are for the estimated target population of 1,034 districts with at 
least one school required to provide SES in the 2004-2005 school year. All 
percentage estimates included in this report have margins of error of plus 
or minus 8 percentage points or less, except for those shown in table 9. All 
other numerical estimates, such as the total number of schools required to 
offer SES in 2004-2005, included in this report have margins of error of 
plus or minus 18 percent or less. 
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Table 9: Sampling Error Calculations 



Page 


Description 


Estimate 


Lower 

bound 


Upper 

bound 


15 


Percentage of districts that spent 20 percent or less of the amount set aside for 
SES in 2004-2005. 


38 


30 


47 


15 


Percentage of districts that spent over 80 percent of the amount set aside for SES 
in 2004-2005. 


18 


11 


26 


17 


Percentage of districts where over half of SES recipients in 2004-2005 were 
elementary school students. 


56 


48 


65 


17 


Percentage of districts where less than 5 percent of SES recipients in 2004-2005 
were students with limited English proficiency. 


33 


22 


43 


17 


Percentage of districts where over half of SES recipients in 2004-2005 were 
students with limited English proficiency. 


21 


13 


32 


17 


Percentage of districts where less than 20 percent of SES recipients in 2004-2005 
were students with disabilities. 


69 


59 


79 


17 


Percentage of districts where over half of SES recipients in 2004-2005 were 
African-American students. 


41 


31 


50 


17 


Percentage of districts where over half of SES recipients in 2004-2005 were 
Hispanic students. 


30 


21 


39 


20 


Percentage of the districts not notifying parents before the beginning of the school 
year that did not receive final school Improvement data from the state prior to 
beginning of the school year. 


51 


42 


61 


20 


Percentage of the districts not notifying parents before the beginning of the school 
year that notified parents within 2 months following the beginning of the school 
year. 


91 


82 


96 


21 


Percentage of rural districts where availability of transportation for students 
attending SES was a moderate to very great challenge. 


50 


37 


63 


32 


Percentage of the districts that observed most or all providers during on-site 
reviews conducted in 2004-2005. 


83 


72 


91 


Source: GAO. 



We took steps to minimize nonsampling errors that are not accounted for 
through statistical tests, like sampling errors. In developing the mail 
survey, we conducted several pretests of draft instruments. We pretested 
the survey instrument with district officials in Woodburn, Ore.; Tacoma, 
Wash.; Baltimore, Md.; and Alexandria, Va., between October and 
November 2005. These pre-tests were similar to the state Web survey pre- 
tests in design and content. On the basis of the pre-tests, the draft survey 
instrument underwent some slight revisions. 

While we did not fully validate specific information that districts reported 
through our survey, we took several steps to ensure that the information 
was sufficiently reliable for the purposes of this report. For example, data 
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from the surveys were double-keyed to ensure data entry accuracy, and 
the information was analyzed using statistical software. After the survey 
was closed, we also made comparisons between select items from our 
survey data and other national-level data sets.^ We found our survey data 
were reasonably consistent with the external sources. On the basis of our 
checks, we believe our survey data are sufficient for the purposes of our 
work. 

Response Rate We received survey responses from 73 percent of the 258 district Title 

I/SES coordinators in our sample. The response rate, adjusted for the 
known and estimated districts that were out of scope, was 77 percent. 



Table 10: SES District Survey Response Rates 


Stratum 


Districts in 
population 


Districts in 
sample 


In-scope 
districts in 
sample 


Districts 

responding 


District enroilment 
>= 100,000 


21 


21 


21 


21 


District enroilment 
< 100,000 


1,074 


237 


227 


167 


Total 


1,095 


258 


248 


188 



Source: GAO. 



After the survey was closed, we analyzed the survey respondents to 
determine if there were any differences between the responding districts, 
the nonresponding districts, and the population. We performed this 
analysis for three characteristics — total number of students enrolled, total 
number of special education students, and total number of English 
language learner students. We determined whether sample-based 
estimates of these characteristics compared favorably with the known 
population values. The population value for all of the characteristics we 
examined fell within the 95 percent confidence intervals for the estimates 
from the survey respondents. On the basis of the 77 percent response rate 



^ We compared our district survey data to data on SES reported by Education in the 
Evaluation of Title I Accountability Systems and School Improvement Efforts: Findings 
From 2002-03 (Washington, D.C.: Nov. 2005) and the National Assessment of Title I: 
Interim Report (Washington, D.C.: Feb. 2006). We also compared our survey data to data 
on SES reported by the Center on Education Policy in From the Capital to the Classroom: 
Year 3 of the No Child Left Behind Act (Washington, D.C.: March 2005) and From the 
Capital to the Classroom: Year 4 of the No Child Left Behind Act (Washington, D.C.: 
March 2006). 
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and this analysis, we chose to include the survey results in our report and 
produce sample-based estimates to the population of districts required to 
provide SES in the 2004-2005 school year. 



Site Visits To understand SES implementation at the local level, we conducted site 

visits to four districts between October 17, 2005, and February 16, 2006. 
The districts visited included Woodburn School District (Woodburn, Ore.), 
Hamilton County Schools (Chattanooga, Tenn.), Newark Public Schools 
(Newark, N.J.), and Chicago Public Schools (Chicago, 111.). The four 
districts visited were selected because they had experience implementing 
SES in their schools and were recommended by stakeholders as having 
promising parent outreach and/or monitoring practices. When viewed as a 
group, the districts also provided variation across characteristics such as 
geographic location, district size, student ethnicity, and the percentage of 
students with limited English proficiency or disabilities. 

During the site visits, we interviewed state officials, including the state 
SES coordinator, and district officials, including the superintendent and 
SES coordinator. We also interviewed officials representing 12 schools, 
including principals and other school staff involved with SES. In total, we 
visited several schools of each level, from elementary to high, and though 
district officials selected the schools we visited, all of the schools had 
experience implementing SES. Through our interviews with state, district, 
and school officials, we collected information on district efforts to notify 
parents and fulfill implementation responsibilities, student participation, 
providers, local implementation challenges, and implementation 
assistance received and needed. During the visits, we also interviewed 
providers and observed tutoring sessions in order to better understand 
implementation. During our visit to Woodburn, Ore., we also observed a 
provider fair. 

In addition to our site visits to four districts, we also visited the Rhode 
Island Department of Elementary and Secondary Education in March 2006 
to gather additional information on state efforts to monitor and evaluate 
SES. Rhode Island invited us to attend two meetings the state held with 
districts implementing SES and providers serving students in Rhode 
Island, during which SES challenges, ways to improve implementation, 
and state efforts to evaluate providers were discussed. 



Provider Interviews in total, we conducted interviews with 22 providers, including 15 providers 

during the site visits and 7 providers operating in multiple states. The 
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Education Industry Association assisted our efforts to contact multi-state 
providers, and most of the multi-state providers we interviewed were 
association members.^ Multi-state provider interviews were conducted 
between November and December 2005. 

Through all of our provider interviews, we collected information on 
provider efforts to increase participation in SES, align services with state 
standards and district curriculum, and communicate with parents and 
schools to ensure students are receiving needed services. We also 
collected information on students served, tutor and program 
characteristics, and provider challenges to SES implementation. While the 
providers we interviewed reflect some variety in provider characteristics, 
our selections were not intended to be representative. Thus, the findings 
from our interviews cannot be used to make inferences about all 
providers. 



Education Data Analyses We analyzed state data submitted to Education through the NCLBA 

Consolidated State Performance Reports (CSPR) for school years 2002- 
2003, 2003-2004, and 2004-2005. State reports from all 3 years included the 
number of students receiving SES and the number of schools those 
students attended, and state reports from 2003-2004 and 2004-2005 also 
included the number of students eligible for SES. Data from the 2003-2004 
CSPRs were used to assist our analysis of SES participation. To assess the 
reliability of the 2003-2004 data, we performed a series of tests, which 
included checking to ensure that data were consistent, that subtotals 
added to overall totals and that data provided for 1 year bore a reasonable 
relationship to the next year’s data and to data reported elsewhere, 
including state education reports. We also spoke with Education officials 
about their follow-up efforts to verify the data. At the time of our review. 
Education was in the process of completing efforts to verify the 2003-2004 
data. 

While we compared the 2004-2005 CSPR data to data obtained through our 
state and district surveys to further verify our data, we generally did not 



^ The seven multi-state providers we spoke with were Babbage Net School, Cambridge 
Educational Services, Catapult Learning, Huntington Learning Center, Newton Learning, 
Platform Learning, and University Instructors. 
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use the 2004-2005 CSPR data for our analysis/ During this comparison 
analysis, where we found discrepancies or sought clarification, we 
followed up with state officials. In several states, officials revised the 
numbers that they had initially reported to us or to Education. On the 
basis of our review of these data, we determined these sources to be 
adequate for the purposes of our work. 

We also considered SES-related findings from Education studies, including 
the Evaluation of Title I Accountability Systems and School 
Improvement Efforts: Eindings Erom 2002-03 (2005) and the National 
Assessment of Title I: Interim Report (2006). To ensure the findings from 
these studies were generally reliable, we reviewed each study’s 
methodology, including data sources and analyses, limitations, and 
conclusions. In addition, in designing our state and district surveys, we 
reviewed SES-related survey questions used by Education in these studies. 



At the time of our survey, seven states were unable to provide 2004-2005 data on SES 
recipients and students eligible to receive SES. By the time Education required states to 
submit the 2004-2005 CSPR reports in March 2006, six of these states had provided this 
data to Education; we, therefore, used this data to supplement their responses to our 
survey. The seventh state. New Jersey, was unable to provide this data to Education by the 
time 2004-2005 CSPRs were due; therefore. New Jersey data is not included in the 2004- 
2005 participation analysis included in this report. 
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UNITED STATES DEPARTMENT OF EDUCATION 

OFFICE OF INNOVATION AND IMPROVEMENT 



July 14, 2006 



Ms. Mamie S. Shaul 

Director, Education, Workforce, and Income Securities Issues 
Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Ms. Shaul: 

I am writing in response to your request for comments on the Government Accountability 
Office (GAO) draft report (GAO-06-758) entitled, “Education Actions Needed to 
Improve Local Implementation and State Evaluation of Supplemental Educational 
Services.” I appreciate the opportunity to respond to the recommendations made in the 
report and to provide you with additional information on how the U.S. Department of 
Education (Department) is supporting effective implementation of the supplemental 
educational services (SES) provisions of Title I of the Elementary and Secondary 
Education Act of 1965, as amended by the No Child Left Behind Act of 2001 (NCLB). 

As you know, the Department takes very seriously the need for effective SES 
implementation at all levels. In a letter to all Chief State School Officers on May 15, 
2006, Secretary Spellings stressed the need for the Department and States to work 
together to expedite the implementation of further improvements of SES. For example, 
we expect districts to notify eligible families about SES in a way that is timely, clear, and 
complete, as well as to spend the full amount of funds necessary to meet demand for SES 
from families. In the May 15 letter, the Secretary also directed States to help their 
districts become fully compliant with SES in 2006-07 by closely monitoring districts’ 
actions and providing them with requisite technical assistance. In such cases where 
districts are not complying with the SES provisions, the Secretary made clear that we are 
fully prepared to take actions such as placing conditions on Title I grants, withholding 
funding, and, if appropriate, entering into compliance agreements. The Department 
believes that these actions, as well as other strategies and actions we are undertaking, 
explained in more detail below, will provide significant motivation and resources to help 
all districts and States be successful with their SES implementation. We appreciate the 
specific recommendations you make in the report and respond to each of them below. 

Your first three recommendations concern additional assistance to help States and 
districts implement SES more effectively, which is a primary goal of the Department and 
one we pursue with full commitment and dedication. Specifically, you recommend that 
the Department: (1) provide better guidance on SES parental notification letters, 

400 MARYLAND AVE., S.W., WASHINGTON, DC 20202 
www.ed.gov 

Our mission is to ensure equal access to educatkm and to promote educational excellence throu^xHit the nation. 
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(2) collect and disseminate information on promising practices used by States to attract 
more providers and used by districts to improve parental notification letters and provider 
services, and (3) provide examples of how districts can involve schools and school 
officials in coordinating activities with providers. 

As you state in your report, the Department has provided States and districts with a 
sample parent notification letter in the June 2005 edition of the SES Non-Regulatory 
Guidance. A team of Department staff, including staff from both the Office of 
Elementary and Secondary Education (OESE) and the Office of Innovation and 
Improvement (Oil), developed this letter, in direct response to the need we saw in the 
field for more informative and higher-quality parent letters. We drafted a sample letter 
not only to contain all required information, but also to be as “parent-friendly” and easy 
to use as possible. However, we recognize that States and districts may need more 
information on parent outreach and communication. The Department has assigned to our 
Comprehensive Center on Innovation and Improvement the task of developing a 
technical assistance effort to help respond to the needs of States, districts, and 
community-based organizations to conduct effective outreach on SES issues. The Center 
will be developing this effort this summer and fall and will implement it in sites around 
the country during the 2006-07 school year. The Center’s effort will include technical 
assistance in the areas of planning and implementing outreach, as well as providing 
sample tools for educators to use to effectively reach parents. Department officials also 
will be meeting with State Title I Directors at the end of July 2006, and in that forum, we 
will solicit further information on the types of technical assistance that would be most 
useful in improving parent outreach and communication, including on how the sample 
letter in the SES guidance might be improved. 

We will also continue to collect and share examples of districts that are effectively 
reaching out to parents and working well with providers. We believe that there are 
lessons to be learned fi'om the districts that participated in the two SES pilots in 2005-06. 
For example, we invited Newport News Public Schools in Virginia to participate in a 
recent production of the Department’s Education News Parents Can Use program that 
focused on SES topics. Newport News shared with a national audience of viewers the 
activities and actions it had taken to implement SES successfully in that district. 

We will also be disseminating promising practices that States use to attract providers 
through our State SES directors meeting, to be held this October. This is the third 
meeting that the Department has sponsored, in conjunction with the Council of Chief 
State School Officers (CCSSO) and with the support of the C.S. Mott Foundation, to 
assist State SES directors with SES implementation. (The meeting also supports 
successful collaboration between SES and the 21“ Century Community Learning Centers 
Program.) As part of this October meeting, we will feature a session on successful State 
actions that help ensure a diversity of providers and on-going, high-quality 
communication between States and providers. During this meeting, we will also feature a 
session exploring successful partnerships between districts, schools, and providers. This 
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session will build on guidance that the Department has already offered in the area of 
school involvement in SES. In the Department’s June 2005 revisions to the SES Non- 
Regulatory Guidance, we encomaged districts to enlist the help of their schools in 
reaching parents. For example, we suggested that districts could use back-to-school 
nights to explain SES to parents and that districts should educate their teachers and 
principals about the SES provisions to make sure they are able to answer parents’ 
questions. We will continue to highlight the important role that schools can play in the 
SES process as we provide technical assistance to the field on SES. 

Your second set of recommendations involves States’ and districts’ ability to fund SES 
sufficiently and provide services to the maximum number of students. As noted above, 
these matters were addressed in various ways in the May 15, 2006, letter. For example, 
the Secretary announced in this letter that the Department was expanding the SES pilot 
program that began last year that allowed four districts in Virginia to provide SES to 
students in schools in year one of improvement. We requested that States interested in 
participating in this program submit proposals by June 19. In the proposals, each State 
had to demonstrate that it met three key conditions related to NCLB implementation: 
timely adequate yearly progress notification, SES evaluation in progress, and State 
assessment systems in one of the top three review categories. States could propose up to 
seven districts to participate in the program. The Department is currently reviewing these 
proposals and anticipates making a decision about this pilot. Additionally, we are 
currently considering expanding the SES pilot that allows districts in need of 
improvement to apply to become SES providers. 

You also recommend that the Department clarify what States can do through the provider 
approval process to set parameters around program design and costs. In the Department’s 
June 2005 revisions to the SES Non-Regulatory Guidance, we provide information to 
States about their role in establishing certain program design criteria for providers to 
meet. We discuss, for example, that a State could establish a range for student/teacher 
ratios or for the rates that providers charge for their programs. We also instruct States 
that they may develop a policy with regard to providers’ use of financial incentives and 
other gifts. We note in the guidance that school districts may not impose requirements on 
providers’ program designs because doing so would undennine States’ authority to 
establish standards for provider approval. While, at this time, the Department has not 
provided additional guidance to alt States on their authority in this area, we continue to 
respond to individual State questions on the matter. Additionally, it is a topic we will 
consider further addressing in our next set of revisions to the SES guidance. 

Your final set of recommendations discusses steps the Department can take to improve 
Federal and State monitoring of SES. The Department continues to improve its 
monitoring efforts of the Title I requirements, including SES. Additionally, as outlined in 
the Secretary’s letter dated May 15, 2006, we urged States to work with local educational 
agencies (LEAs) in their efforts to comply with SES requirements and to closely monitor 
the LEAs’ actions in this area. 
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With regard to this recommendation, you specifically recommend that the Department 
require States to report information necessary to determine the percentage and amount of 
Title I funds spent by districts to provide SES. The Department recognizes the 
importance of collecting information on how much districts are spending on SES (as well 
as how much they are spending on public school choice-related transportation). In the 
2006-07 data collection for the Education Data Exchange Network (EDEN)/EDE'actj 
system, the Department is proposing that districts report on their SES and public school 
choice spending. This will give the Department data on spending trends across the 
country, as well as help us identify districts that are doing either particularly well, or are 
under-spending, on SES. 

Finally, you recommend providing States with technical assistance on how to evaluate 
the effects of SES on student achievement. The Department has recognized the 
importance of high-quality State evaluations for several years. As part of a grant to the 
American Institutes for Research, the Department supported the development of an issue 
brief to advise States on the evaluation process. (This issue brief is available at: 
http://www.tutorsforkids.org/documents/SES Evaluation Issue Brief 004.pdf ). The 
Department and the authors of this issue brief have shared it widely and presented it at 
several national conferences. Additionally, the Department has assigned the 
Comprehensive Center on Innovation and Improvement the task of providing additional 
assistance in this area. The Center will work with the authors of the original issue brief to 
update it by this September and disseminate it to the field, and the Center has hired the 
authors as technical advisors so that they can provide ongoing technical assistance to 
States in the area of evaluation. Finally, this fall at the Department’s conference with the 
CCSSO and the C.S. Mott Foundation, we will hold several in-depth sessions on 
conducting evaluations to provide more technical assistance to States. 

Thank you again for taking the time to research and report on the SES provisions of 
NCLB. The Department values the work that you have done to provide rich and 
compelling information about the current status of SES around the country. We will use 
the findings and recommendations made in this report to improve our technical assistance 
to States and districts and ultimately to improve the quality of SES implementation 
around the country. 

Sincerely, 

Christopher I. Doherty 

Acting Assistant Deputy Secretary 

Office of Innovation and Improvement 
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